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ABSTRACT 

A  current-technology  computing  machine  may  he  roughly  decomposed  Into  a 
processor,  a  memory,  and  a  data  path  connecting  therm  The  interposition  of  this 
data  path  between  processing  and  storage  elements  creates  a  bottleneck,  which 
inhibits  progress  at  the  high-performance  end  of  the  technological  spectrum.  Addi¬ 
tionally.  the  monolithic  nature  of  present-day  processors  resists  incremental  addi¬ 
tion  or  removal  of  processing  power 

The  research  described  bore  attacks  the  problem  of  constructing  more  power¬ 
ful  and  more  flexible  computer  systems  along  throe  fronts  the  definition  of  a  vir¬ 
tual  machine  providing  for  parallel  computation  using  ob/ects  and  object  references, 
the  development  of  a  distributed  implementation  mechanism  ("reference  trees") 
supporting  object  management  functions  including  garbage  collection,  and  the  inves- 
tiga  tion  of  scheduling  algorithms  and  collection  of  performance  results. 

A  reference  free  network  using  these  concepts  Is  composed  of  a  multitude  of 
independent  small  processors,  yet  operates  as  a  coherent  programming  system 
Programs  and  data  spread  automatically  and  transparently  through  the  network  to 
occupy  underused  resources  The  modular  structure  of  the  network  provides  many 
parallel  data  paths  as  well  as  allowing  for  easy  addition  or  removal  of  modules,  thus 
addressing  some  of  the  problems  discussed  above  A  prototype  reference  tree 
network,  the  MuNef,  is  currently  in  operation 
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Chapter  1:  Introduction 


A  good  part  of  the  history  of  computer  science  has  been  the  story  of  man’s 
struggle  with  the  capacity  of  his  computing  machinery.  At  one  extreme  lies  his 
drive  to  construct  ever  more  powerful  machines,  at  almost  any  cost.  Within  this 
expanding  limit  of  technology,  man  faces  the  constantly  changing  demands  on  his 
computer  systems,  trying  to  adapt  yesterday's  computing  structures  to  face 
tomorrow's  challenger 

Both  quests  are  hampered  by  the  relatively  monolithic  nature  of  today's  com¬ 
puter  systems  A  current-technology  computing  machine  may  be  roughly  decom¬ 
posed  into  a  processor,  a  memory,  and  a  data  path  connecting  them  The  interposi¬ 
tion  of  this  data  path  between  processing  and  storage  elements  creates  a  "von 
Neumann  bottleneck"!  1  ]  which  inhibits  progress  at  the  high-performance  end  of  the 
technological  spectrum 

Even  at  lower  points  on  the  spectrum,  the  bottleneck  causes  problems 
A'though  memory  si/e  can  be  scaled  reasonably  successfully  to  match  user  require¬ 
ments,  the  monolithic  nature  of  the  processor  defies  incremental  modification  The 
best  approach  to  this  problem  that  is  currently  widely  available  is  the  concept  of 
compatible  families  of  processors  Using  this  approach,  when  a  processor's  capa¬ 
city  becomes  inappropriate,  it  may  be  replaced  by  a  more  suitable  compatible  proc¬ 
essor  capable  of  executing  the  same  software,  thus  avoiding  the  overhead  of 
starting  completely  over  from  scratch. 

One  imaginable  approach  to  giving  processors  more  of  the  scaling  properties  of 
memories  is  to  levrlop  processors  with  multiple  functional  units  Thus,  at  least  in 
theory,  a  computer's  processing  capacity  could  be  altered  by  the  simple  addition  or 
removal  of  functional  units  This  approach  has  a  couple  of  drawbacks.  First, 
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Commerc  lally  available  processors  with  multiply  functional  units  stil1  attempt  to  give 


the  appearance  of  executing  a  single  sequential  instruction  stream,  and  this  seman¬ 
tic  constraint  limits  the  number  of  functional  units  that  can  be  used  effectively. 
Second,  even  If  this  constraint  is  removed,  the  data-path  bottleneck  limits  the 
number  of  functional  units  that  can  be  kept  busy  In  essence,  as  more  processing 
power  and  memory  are  added  on  opposite  sides  of  the  bottleneck,  the  effective 
usage  rate  ot  a  unit  of  either  must  drop.  The  point  at  wtiich  this  begins  to  happen 
is  determined  by  the  bandwidth  of  the  bottleneck 

The  C  mmp  project[40.41  ].  therefore,  attempted  to  increase  the  bandwidth  of 
the  bottleneck  by  creating  many  parallel  data  paths  between  the  bank  of  process¬ 
ing  units  and  the  bank  of  storage  units  Still  intact  in  C  mmp  is  the  principle  that 
every  processing  unit  should  be  able  to  access  every  item  in  storage  with  equal 
ease  The  programming  environment  presented  by  C  mmp  is  roughly  one  in  which  a 
number  of  concurrent  processing  units,  share  a  common  memory  not  too  different 
from  the  virtual  machine  presented  by  many  of  today's  operating  systems  If  a 
user  wishes  to  obtain  the  full  potential  throughput  of  the  system,  he  must  break  his 
job  into  enough  parallel  tasks  to  keep  all  the  processors  occupied  Means  for 
manually  doing  this  am  well  known,  if  perhaps  not  all  that  easy  to  work  with  The 
user  is  not  forced  to  adopt  any  bizarre  programming  style,  or  face  any  situation 
where  data  is  not  equally  accessible  to  all  tasks  The  drawback  of  C.mmp  is  that 
its  widening  of  the  processor-memory  data  path  is  expensive  Its  10x16  crossbar 
switch  is  a  large  piece  of  hardware,  and  engineering  problems  almost  prevented  it 
from  ever  workirg.  Thus  it  is  not  clear  how  far  that  approach  can  be  scaled  up. 
f  ven  if  it  can,  its  si/e  grows  as  the  product  of  the  number  of  processing  and 
memory  units. 

It  is  possible  to  generate  a  strongly  connected  (every  memory  unit  equally 
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iicressible  to  every  p'  ?ssing  unit)  data  pa’h  wdhou*  mourning  quadratic  expan¬ 
sion  costs  —  Batcher  sorting  nets[4]  and  the  arbitration  and  distribution  nets  from 
data  flow  computers[Y,30]  are  examples  of  such  organisations,  however,  the  cost 
always  grows  more  than  linearly  in  the  number  of  processing  and  memory  units  An 
extreme  reaction  to  this  situation  is  to  directly  connect  each  processing  unit  to 
only  its  own  private  memory  unit,  and  arrange  some  other  mechanism  for  proces  >or- 
memory  pairs  to  exchange  data  with  each  other  This  approach  is  evident  in  the 
DCS  ring  network[9, 1 0],  ARPANF  T[?0],  [  thernetj  ?  7  ].  and  countless  other  ad  hoc 
telecommunications  networks  that  exist  Such  network  >  achieve  a  form  ot  scalabil¬ 
ity  in  that  processor-memory  pairs  may  be  added  or  removed  relatively  easily  in 
most  cases  However,  there  is  very  often  a  central  medium  which  can  only  physi¬ 
cally  support  a  certain  number  of  connections,  and  which  can  also  cause  a 
bandwidth  bottleneck  if  large  amounts  of  communication  are  required 

Perhaps  more  importantly,  such  networks  often  do  not  form  coherent  systems 
with  good  support  for  co-ordinated  parallel  processing  Indeed,  this  is  not  usually 
their  purpose  they  exist  instead  as  vehicles  for  sharing  information  present  at 
the  various  nodes,  and  accomplish  even  that  by  often  arcane  and  special-purpose 
mechanisms  This  detracts  from  their  usefulness  as  solutions  to  the  problem  of  the 
von  Neumann  bottleneck,  limiting  it  to  cases  where  many  small  and  largely  self- 
contained  tasks  are  being  run 

A  compromise  between  these  weakly  connected  networks  just  discussed  and 
structures  of  the  C  mmp  class  is  the  C m*  oroject[3f>]  This  project  avoids  some  ot 
the  problems  of  a  central  communications  medium  by  attaching  its  processor-memory 
pairs  as  leaves  ol  "ree'ike  structure  of  "clusters  "  Any  processor  can  still 
address  any  word  of  memory  in  the  system,  but  the  access  time,  though  still  fairly 
short,  increases  with  the  length  of  the  path  through  the  tree  to  the  desired  word 
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As  a  result  ot  this  abandonment  of  the  "equal  access"  philosophy,  Cm-  is  able  to 
achieve  linear  scaling  behavior  1  he  basic  semantics  of  a  shared-memory  system 
are  preserved,  although  there  is  some  incentive  for  the  user  to  arrange  for  compu¬ 
tations  to  be  performed  near  the  data  they  reference  Two  objections  can  be 
made  to  Mte  Cm-  ap>proach  first,  its  scalability  may  still  bo  limited,  although  the 
bound  is  probably  loose  enough  to  be  primarily  of  theoretical  interest  there  is  a 
set  of  "central"  media  that  handle  accesses  crossing  cluster  boundaries;  as  the 
number  of  clusters  grows,  those  media  could  become  saturated 

Second,  Cm-  is  un  expensive  way  of  connecting  processing  elements  As  jiroc- 
essors  and  memories  become  smaller  and  cheaper,  they  may  be  dwarfed  by  the 
interconnection  hardware  I  ho  cost  ot  communication  hardware  will  probably  drop 
along  with  the  cost  of  other  components,  but  the  ratio  of  communication  to  comput 
ing  hardware  in  Cm-  is  still  quite  large  A  principal  reason  tor  the  cost  ot  tins 
hardware  is  the  demand  tor  real-time  performance  placed  on  it  by  the  architecture 
ot  Cm-  when  a  processor  requests  a  non-local  memory  access,  it  is  prevented 
from  continuing  until  the  access  lias  been  performed.  A  system  that  did  not  make 
this  demand  would  ho  able  to  use  communication  hardware  chosen  from  a  wider 
range  on  the  performance  spectrum 

That  the  Cm-  system  penalizes  nonlocal  memory  accesses  by  a  processor,  rt>la- 
tive  to  local  references,  implies  that  optimal  performance  by  the  network  depends 
on  a  certain  amount  of  locality  among  the  tasks  being  executed  on  the  network 
the  system  will  work  poorly  if  memory  accesses  made  by  each  processor  are  uni¬ 
formly  distributed  throughout  the  entire  memory  of  the  system  rfut  it  locality  of 
reference  is,  or  an  tie  mao»,  a  strong  enough  effect,  perhaps  the  penalty  on  non¬ 
local  accesses  could  even  be  increased  without  seriously  degrading  system  perfor¬ 
mance.  If  a  fairly  high  penn  tv  is  acceptable,  it  should  be  feasible  to  get  by  with 
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much  less  exotic  interconnection  hardware  than  in  Cm*  this,  in  tact,  is  the  theory 
behind  the  loosely  coupled  networks  discussed  earlier  Ihe  tasks  processed  by 
these  networks  display  a  great  deal  of  locality,  internode  communications  are  rela¬ 
tively  expensive  Yet  these  networks  often  do  their  jobs  quite  well  Ihe 
difference  is  tfiat  Cm*  at  least  supports  some  kind  of  coherent  methodology  for 
making  nonlocal  accesses,  while  many  networks  offer  none  at  all 

1.1:  The  MuNet 

It  seems  plausible  and  useful,  then,  to  consider  constructing  a  network  which 
is  tailored  for  tasks  that  exhibit  some  locality  of  reference,  but  which  still  supports 
a  coherent,  transparent  methodology  for  making  nonlocal  accesses  to  data  any¬ 
where  in  the  system  Virtually  any  *  urrent  networking  scheme  can  probably  hr* 
used  as  a  ha  <e  tor  such  a  system  the  key  element  is  the  environment  provided 
cn  each  processc  •  in  the  network  for  ttu>  execution  of  user  software  The  provi¬ 
sion  of  such  n  software  environment,  which  allows  transparent  access  to,  and 
mot. on  of  data  objects  throughout  the  system,  r  a  central  part  of  the  MuNet  proj¬ 
ect  13,1  4  1  6,34.35.3  311]  The  MuNet  is  n  network  built  for  the  purpose  of  test¬ 

ing  the  various  ideas  reported  in  this  thesis  Its  hardware  is  composed  of  several 
l  SI- 1  1  processors.  ea<  h  with  ?8k  words  of  private  memory  I  ach  processor  has 
ports  for  up  to  four  125Miaud  serial  lines  which  an  lie  connected  to  other  proces 
sors  to  create  a  network  configuration  of  some  desired  topology  Tin*  design  of  the 
MuNet  hardware  's  not  hailed  as  any  advance  in  technology  for  constructing  com 
pin  >r  networks,  quite  to  the  contrary,  it  is  simple  in  the  extreme,  since  the  princi¬ 
pal  goal  of  the  MuNet  project  is  the  development  of  software  methodologies  and 
organizational  principles  for  using  such  networks 
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1 The  Virtual  Machine 


We  begin  in  Chapter  ?  of  this  thesis  by  considering  the  virtual  machine,  or 
Interface,  through  which  programs  can  invoke  internal  network  machinations,  only 
later  are  the  machinations  themselves  examined  This  virtual  machine  is  an 
abstracted  and  improved  version  of  the  environment  provided  for  program  execution 
on  the  MuNet  and  has  its  conceptual  roots  in  earlier  work  on  the  "mu  calculus” 
message-passing  formalism[  1  f>,3H  ]  A  detailed  description  of  the  MuNet  environ¬ 
ment.  along  with  a  commentary  on  It  appears  in  Appendix  A  Occasions  to  teter  to 
the  virtual  machine  described  in  Chapter  ?  will  arise  frequently,  so  we  succumb  to 
the  temptation  to  coin  an  acronym,  and  name  it  VIM  (Virtual  /nterfoce  to  the 
MuNet)  Ihis  term  should  be  understood  to  mean  specifically  the  virtual  machine 
outlined  in  Chapter  ?,  not  the  closely  related  one  Described  in  Appendix  A  In 
broad  outline  VIM  is  a  garbage-collected  object-reference  system[fi]  which  sup¬ 
ports  message-passing,  or  actor-style,  computation  Our  treatment  of  VIM  is  at  two 
'evels  first,  an  introduction,  writtf<n  in  f  nglish  prose,  and  second,  an  informal 
"h'a.  kboartf  interpreter."  akin  to  the  "contour  model"  used  for  block  structured 
language*-,  to  make  the  specification  more  precise 

VIM  is  important  in  that  it  serves  as  i  c'r>an  interface  between  the  user's 
requirement s  and  the  system's  capabilities,  but  its  specification  only  begins  to 
attack  the  problem  of  constructing  a  viable  network  Nevertheless,  the  choice  of 
this  starting  point  underscores  a  ley  principle  behind  this  thesis  research  that 
the  environment  provided  for  program  execution  is  a  much  more  significant  aspect 
of  a  system  than  the  particular  technology  of  its  construction 
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1.3:  The  Physical  Machine 


l  very  virtual  machine  must  bo  supported  by  some  underlying  implementation, 
and  an  early  hurdle  that  must  be  passed  by  any  practical  scheme  is  the  demonstra¬ 
tion  that  there  exists  a  viable  implementation  of  it  details  ot  an  implementation  of 
VIM  will  obviously  be  influenced  by  details  of  the  structure  of  the  underlying  net¬ 
work  f  or  concreteness,  then,  we  are  forced  to  make  choices,  some  of  them  fairly 
arbitrary,  about  those  details  We  pck  a  network  of  identical  processors,  each 
with  some  amount  of  private  memory  attached  I  ach  processor  communicates 
directly  with  only  a  limited  number  of  other  processors  (or;  four)  in  the  network 
There  are  two  reasons  for  making  this  choice  first,  it  lends  itself  to  extremely 
simple  communication  hardware,  keeping  communication  costs  in  proportion  to  the 
costs  of  other  parts  of  tht  netwo'k  Second,  it  avoids  scaling  problems  the 
absence  of  any  central  medium  means  the  absence  of  any  opportunity  to  saturate 
it,  hence  a  processor  array  of  our  design  could  in  principle  be  iterated  to  arbitrarily 
large  si/e 

It  can  ho  argued  that  as  our  network  is  expanded,  the  distance,  and  ti€»nce 
delay,  of  nonlocal  act. esses  will  increase,  defeating  the  scalability  arqument  This 
is  true  if  accesses  by  a  processor  are  uniformly  distributed  over  the  network 
However,  if  tasks  exhibit  a  suitable  locality  o,  reference,  this  need  not  be  sc 
T  xpanding  the  network  will  always  increase  its  capacity  to  handle  more  tasks  con¬ 
currently,  t trough  at  some  point  it  will  i  ease  to  affect  the  real  time  required  to  com¬ 
plete  any  narticula'  task  at  that  point,  the  task  will  he  spread  thinly  enough  that 
any  further  spreading  w  iuM  increase  communication  delays  more  than  warranted  by 
the  processing  power  game d  It  i  possible,  however,  to  construct  networks,  such 
as  binary  trees,  where  the  power  gained  is  exponential  in  the  increasing  distance 
between  the  fa'  hest-apart  n<  dcs  serving  a  task  (The  three-dimensionality  of 
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space  does  put  an  ultimate  limit  on  the  application  of  this  approach,  hut  only  at  the 
point  where  the  length  of  wires  connecting  nodes  becomes  significant^/ J  ) 

It  may  be  that  a  central  medium  is  actually  more  cost-effective,  up  to  some 
size,  than  the  fully  decentralized  scheme  proposed  here  As  such  a  network  with  a 
central  medium  is  scaled  up,  though,  a  time  will  come  when  it  is  impossible  to  con¬ 
tinue  connecting  processors  to  that  central  medium  At  this  point,  the  central 
medium  will  have  to  be  split  into  two  "central"  media,  with  some  link  between  them 
A f t o r  further  size  increases  occur,  a  macroscopic  view  of  the  resulting  network  will 
look  much  like  our  proposal  a  collection  of  tightly  coupled  processor-memory  units 
(one  such  unit  corresponding  to  each  "central  medium")  connected  so  that  each 
unit  diroctly  communicates  with  sumo  number  of  neighboring  units  I  bus  although 
our  approach  may  not  be  tin*  best  for  some  smal1  networks,  it  deals  with  the  limiting 
case  of  any  network  as  it  is  scaled  up 

Iho  ultimate  validity  of  our  approach  stems  from  the  fact  that  it  supports  an 
appropriate  environment  for  program  execution,  but  its  usefulness  depends  on  the 
practical  viability  of  an  implementation  the  most  basic  problem  here  is  the 
management  of  data  and  code  objects,  to  allow  nonlocal  accesses,  motion  of 
objects  from  one  processor  to  ano*her.  and  garbage  collection  across  the  network 
These  functions  are  accomplished  using  data  structures  known  as  reference 
frews[  1 5, 1 6],  leading  any  interconnection  of  processors  using  this  scheme  to  be 
dubbed  a  rmfpronct'  free  rn^t\A/orh  C  hapter  6  describes  the  use  of  reference  trees 
to  support  the  va-ious  operations  required  by  VIM 
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1.4:  System  Performance 

Reference  trees  allow  the  basic  network  functions  to  be  performed  However, 
a  network  will  not  operate  up  to  its  potential  unless  good  decisions  are  made  about 
how  to  allocate  tasks  and  data  among  processors 

Inefficiencies  in  the  MuNet  implementation  fall  into  two  categories  One 
category  includes  overhead  incurred  during  normal  housekeeping  chores  inside  proc¬ 
essors,  such  as  free  storage  management  and  task  enqueuing  and  dequeuing,  the 
Other  Includes  lost  time  resulting  from  unfortunate  distribution  of  tasks  or  data,  or 
from  non-optima  I  reference  tree  configurations  The  former  kind  of  inefficiency  might 
be  colled  "tactical."  the  latter  "strategic”  Tactical  inefficiencies  may  b»  remedied 
by  more  careful  coding  or  more  appropriate  hardware,  but  do  not  present  research 
questions  along  the  general  thrust  of  the  thesis  Tor  the  most  part,  they  relate  to 
activities  ttiat  would  he  required  on  any  garbage-collected  message-passing  sys¬ 
tem,  even  a  single-processor  one  Strategic  inefficiencies,  on  the  other  hand,  are 
Intimately  related  to  strategies  of  operation  of  a  particular  network  Furthermore, 
strategic  inefficiencies  can  have  effects  which  are  irders  of  magnitude  greater 
than  those  of  tactical  inefficiencies,  and  also  quite  possibly  grow  faster  than 
linearly  in  the  I'/r  of  the  network 

for  these  reasons,  we  shall  he  primarily  concerned  with  inefficiencies  of  the 
strategi  kind  Our  principal  metric  will  not  tie  wnether  the  network  performs  better 
at  some  task  than  another  configuration  o'  simi  ar  i.ost  could,  given  comparable 
efforts  devoted  to  programming  the  task  in  suitable  ways  for  each  configuration 
(although  this  woo'd  be  a  reasonable  way  to  judge  a  commercial  product)  Rather, 
we  will  compare  the  performance  of  similar  m^ssaqr  passing  programs  (r.e..  subject 
to  the  same  tactical  inefficiencies)  and  see  if  employing  the  network  brings  an 
acceptable  l”crease  in  performance  over  that  afforded  by  a  s’ngle  jirocessor  or 
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smaller  network  the  scheduling  algorithms  described  in  this  thesis  do  exhibit  rea¬ 
sonable  scaling  behavior  over  different  sizes  of  networks,  at  least  for  some  applica¬ 
tions 

1.5:  Summary 

T’resent-day  machines  do  not  scale  well  This  is  because  of  the  monolithic 
nature  of  current  processors,  and  (in  the  high-performance  realm)  because  of  the 
"von  Neumann  bottleneck"  separating  processor  from  memory  Attempts  to  "widen" 
the  bottleneck  by  adding  parallel  paths  within  it  are  unattractive  because  the  com¬ 
plexity  of  the  hardware  involved  increases  more  than  linearly  in  the  degree  of 
widening  achieved  An  alternative  is  to  replace  a  big  von  Neumann  bottleneck  with 
several  independent  little  von  Neumann  bottlenecks,  in  the  form  of  small  processor- 
memory  pairs  connected  In  a  network  This  alternative,  which  is  a  way  of  distribut¬ 
ing  processing  power  among  the  memory  (or  vice  versa),  has  the  advantage  of 
being  scalable  to  arbitrarily  large  sizes  If  the  network  nodes  are  sufficiently  small 
and  numerous,  the  network  solution  may  also  ofler  an  answer  to  our  first  complaint, 
about  the  difficulty  of  making  fine  adjustments  to  the  capacity  of  today's  proces¬ 
sors  Given  the  current  economics  of  1  SI  chip  technology,  it  may  also  offer  i  more 
inexpensive  way  to  deliver  computing  power 

The  utility  of  such  a  network  will  ho  seriously  compromised,  however,  without 
nn  appropriate  semantic  coherence  among  the  nodes  and  reasonable  transparency 
of  network  operations  This  thesis  reports  on  the  specification  of  a  standard 
environment  for  program  execution  which  has  these  desirable  propert'es.  and  the 
design  and  evaluation  of  an  implementation 

The  problem  of  constructing  scalable,  high-performance  systems  is  thus 
attacked  along  three  fronts  the  definition  of  the  VIM  virtual  machine  the 
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development  of  the  reference  tree  mechanism  for  object  management,  and  the 
investigation  of  scheduling  algorithms  and  collection  of  performance  results.  These 
efforts,  although  they  all  come  together  in  the  architecture  of  reference  tree  net¬ 
works,  are  nevertheless  to  some  extent  independent  In  particular,  the  VIM  virtual 
mai  hine  doe-,  r of  presuppose  in  any  way  an  implementation  using  reference  trees, 
and  should  be  suitable  in  any  context  where  an  object-oriented  system  able  to  han¬ 
dle  concurrency  is  desired  Conversely,  the  reference  tree  scheme  could  be  used 
to  support  a  wide  variety  of  virtual  machines  needing  Its  abilities  to  deal  with 
obje  ts  an  !  object  refereri.i  ,  and  to  perform  garbage  collection  and  synchroniza¬ 
tion  of  access  to  objei  ts  VIV  and  reference  trees  are  a  good  match  for  each 
other,  though,  and  come  together  in  the  evaluation  of  performance  results  In 
Chapter  7 

1.6:  Thesis  Overview 

this  thesis  has  eight  chapters  Chapter  1  is  this  introduction  Chapter  2 
describes  the  basic  VIM  virtua1  machine  and  its  blackboard  interpreter  Chapter  3 
outline,  some  possible  oxtens  on.s  to  VIM  to  handle  demands  that  might  be  made  by 
opernt-np  *-ystems  running  ur-der  VIM  Chapter  4  describes  the  kinds  of  physical 
networks  on  whir  h  the  proposed  implementations  of  VIM  are  to  he  constructed 
The  "refereri  ,o  tree”  mechanism  at  the  heart  of  these  implementations  Is 
presr  i.tod  in  Chapter  f>  Chapter  R  is  a  discussion  of  network  performance. 
Chapter  7  deal  v:th  st-ategu  'or  distributing  objects  and  tasks  around  a  net¬ 
work,  and  Chapter  8  contams  concluding  thoughts 
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2.1:  Introduction  to  VIM 

Programs  running  under  VIM  belong  to  a  universe  populated  with  two  basic 
kinds  of  entities  objects  and  events  Data  and  algorithms  are  represented  as 
objects.  which  serve  functions  similar  to  those  of  flits  In  operating  systems,  or 
c«// s  m  LISP[?5]  Computations  to  bo  performed  are  represented  as  events,  which 
correspond  very  roughly  to  processes  In  traditional  operating  systems  The  con¬ 
cepts  of  object  and  event  derive  from  research  into  actor  systems[3. 1  8.23] 

The  information  contained  in  an  object  is  recorded  in  its  fe»f  A  text  may  be 
represented  as  a  series  of  words  In  memory,  together  with  some  format  Information 
The  format  information  indicates  the  length  of  the  text,  and  also  identifies  certain 
of  tho  words  as  refer  ences[5]  to  other  objects  (or  even  to  the  same  object 
circular  structures  are  perfectly  legitimate)  The  remaining  words  need  not  be 
intelligible  to  VIM  and  may  contain  arbitrary  binary  data  The  space  of  objects  is 
assumed  to  be  garbage-collected,  so  no  explicit  de  allocation  of  objects  need  ever 
occur  (In  fact,  VIM  provides  no  facility  for  explicitly  de-allocating  objects  ) 

In  the  blackboard  interpreter,  an  object  is  represented  as  shown  in  figure  2.1 
Each  object  is  drawn  as  a  series  of  slots  which  represent  the  contents  of  that 
object's  text  and  bear  symbolic  names  (for  convenience  in  using  tho  blackboard 
interpreter)  The  names  may  be  integers,  such  ns  0  and  1.  ordinary  identifiers, 
such  as  a  or  b,  or  special  reserved  symbols  such  as  9  Each  slot  may  contain  a 
reference  to  another  object,  drawn  as  an  arrow  originating  in  the  slot  and  pointing 
At  the  object  referenced  (as  in  tho  slots  9  and  b  of  Figure  P  I).  A  slot  may  alter¬ 
natively  contain  a  special  null  reference  drawn  as  a  diagonal  line  (as  in  slot  0  of 
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Figure  2  1:  Blackboard  representation  of  an  object 
Figure  2.1),  or  an  arbitrary  Integer. 

Another  feature  of  every  object  Is  an  end  marker,  drawn  as  a  double  vertical 
line,  which  indicates  the  end  of  the  object's  text  Only  the  contents  of  slots  to 
the  left  of  an  object's  end  marker  are  accessible  to  executing  programs  (Exam¬ 
ples  will  be  given  later  in  which  there  are  slots  to  the  right  of  the  end  marker;  for 
now,  that  possibility  may  be  ignored.)  Tho  number  of  slots  from  the  beginning  of  an 
object  to  Its  end  marker  is  referred  to  as  the  object's  length 

In  form,  an  event  resembles  an  object  text  It  too  may  bo  represented  as  a 
series  of  words  in  memory  with  associated  format  information,  and  it  too  may  con¬ 
tain  both  object  references  and  uninterpreted  words  A  particular  slot  within  each 
event,  however,  is  roserved  for  a  reference  to  its  target  object  Semantically,  an 
event  Is  a  request  to  execute  code  associated  (in  a  way  to  be  described)  with 
the  target  object  During  execution  of  that  code,  the  references  and  words  con¬ 
tained  in  the  event  will  be  available  as  parameters  Upon  completion  of  execution, 
Its  mission  fulfilled,  an  event  is  deactivated  (and  presumably  reclaimed  by  the 
storage  system).  Thus  if  further  computations  ought  to  ensue,  execution  of  an 
event  should  result  In  the  creation  of  re w  events  requesting  the  performance  of 
those  computations  It  Is  possible  to  "fork  off"  several  consequent  events  from  a 
single  event  execution,  and  it  is  possible  to  construct  "join"  operators  that  wait  for 
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the  completion  ot  several  events  betore  continuing,  indeed,  it  is  expected  that 
the  use  of  such  parallelism  will  help  greatly  in  making  the  most  effective  use  of  a 
network  Within  a  single  thread  of  events,  however,  there  is  only  the  simple  con¬ 
trol  structure  ot  event  creation  Subroutine  calls  and  returns,  loops,  and  other 
more  complicated  control  structures  may  be  implemented  using  co/if /ni/«if/ons[  1  d] 

An  event  in  the  blackboard  interpreter  is  drawn  as  in  Hgure  2  2.  As  in  the 
case  of  an  object,  an  event  is  drawn  as  a  series  of  named  slots,  each  of  which 
may  contain  an  object  reference  (note  that  there  is  no  such  thing  as  a  reference 
to  an  event),  a  null  reference,  or  an  integer  The  reserved  identifier  r  marks  the 

slot  containing  the  reference  to  the  event's  target  object.  A  special  triangular 

area  is  drawn  to  the  right  of  the  end  marker,  containing  a  Boolean  active  flag.  The 
value  of  this  flag  is  true,  or  T.  if  the  event  should  be  executed,  and  otherwise  is 

false,  or  f  This  latter  state  can  occur  if  the  event  in  question  has  already  been 

executed,  or  for  other  reasons 


end  marker 


figure  2  2  Blackboard  representation  of  an  event 


In  the  course  of  executing  an  event,  several  special  services  provided  by  VIM 
may  bo  Invoked  It  is  often  necessary  to  examine  and/or  modify  the  text  of  an 
object,  so  VIM  offers  facilities  for  gaming  access  to  a  text  in  either  read-only  mode 
(sharable  with  other  readers)  or  read/write  mode  (sharable  with  no  one)  Request¬ 
ing  either  service  entails  specifying  to  VIM  a  reference  to  the  desired  object 
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Execution  of  the  requesting  event  is  not  allowed  to  continue  until  access  in  the 
indicated  mode  is  acceptable  Once  granted,  permission  to  access  a  text  in  a  cer¬ 
tain  mode  persists  during  the  remainder  of  the  execution  of  the  event.  Thus  if  an 
event  has  acquired  non-shared  access  to  an  object,  no  other  event  execution  may 
access  the  object  while  the  original  event  continues  to  execute.  These  permis¬ 
sions  are  not  Inherited  by  any  subsequent  events,  however,  which  must  re¬ 
establish  (via  new  calls  to  VIM)  any  access  privileges  they  desire  ’ 

The  VIM  style  of  access  to  object  texts,  analogous  to  locking  protocols  used  in 
transaction-based  database  management  systems[8,1  1 .32],  makes  possible  the 
construction  of  various  kinds  of  synchronization  operators  without  additional  special 
provision  within  the  VIM  implementation  itself  Operators  such  as  semaphores  need 
to  examine,  and  possibly  modify,  object  texts  free  from  inconsistencies  caused  by 
concurrent  activities  of  other  processes,  and  without  causing  any  other  process  to 
see  any  inconsistencies  Ry  controlling  the  areas  of  possible  interference  between 
concurrent  event  executions,  the  VIM  access  style  simplifies  the  construction  and 
understanding  of  programs 


*lt  is  important  to  note  that  this  regulation  of  "access  privileges"  is  not 
intended  as  a  protection  mechanism  Sigm*lcant  protection  may  inhere  In  the  fact 
that  it  is  impossible  for  an  event  to  access  the  text  of  any  object  to  which  it  can¬ 
not  obtain  a  reference,  but  this  protection  is  insufficient  in  many  cases  Additional 
protection  must  be  supplied  either  through  higher-level  constraints  (e.g.,  forced  use 
of  some  high-level  language  compiler)  or  through  augmentations  to  the  facilities  pro¬ 
vided  by  VIM  The  use  of  either  alternative  as  a  protection  mechanism  is  only 
tangentially  within  the  scope  of  this  thesis 
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2.2:  Deadlock  Avoidance 


Ihe  VIM  scheme  for  obtaining  non-shared  access  to  object  texts  sounds  like  a 
sure  recipe  for  deadlock  any  time  concurrently  executing  events  attempt  to  gain 
such  access  to  the  same  objects  In  order  to  be  able  to  resolve  deadlocks.  VIM 
must  have  the  authority  to  abort  an  event  execution  any  time  additional  access 
privileges  are  requested  (In  fact.  VIM's  authority  to  abort  events  is  slightly 
broader  than  this  )  Aborting  the  execution  of  an  event  frees  any  resources  (such 
as  access  to  object  texts)  that  the  event  might  have  accumulated,  allowing  other 
events  waiting  for  those  resources  to  proceed  Execution  of  the  aborted  event 
may  then  be  tried  again  later 

When  VIM  aborts  an  event,  it  should  create  the  appearance  that  execution  of 
the  aborted  event  was  never  attempted  If  this  is  done,  programmers  working 
under  VIM  will  never  have  to  consider  the  possible  influence  of  previously  aborted 
executions  on  the  final,  successful  execution  of  an  event  Therefore,  if  an  aborted 
event  had  performed  any  modifications  to  object  texts,  those  modifications  must  bo 
undone  This  problem  can  be  avoided  by  prohibiting  an  event  from  making  requests 
for  additional  resources  once  it  has  performed  any  such  modifications.  It  all  events 
observe  this  discipline,  no  event  that  is  aborted  will  ever  have  performed  any  side 
effects,  and  events  can  be  aborted  correctly  without  adjusting  the  contents  of  any 
object  texts  Accordingly,  we  modify  our  previous  statement  concerning  VIM's 
authority  to  abort  events,  stipulating  that  execution  of  an  event  may  be  aborted 
only  if  it  has  performed  no  side  effects  (the  concept  of  "side  effect"  will  be  made 
more  precise  in  the  definition  of  the  blackboard  intorpieter).  So  that  an  event  can 
be  aborted  any  time  it  requests  additional  access  privileges,  we  must  require  that 
such  requests  be  mede  only  when  the  requesting  event  has  not  yet  performed  any 
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side  effects 


Requiring  events  to  adhere  to  this  discipline  does  not  reduce  the  logical  power 
ot  VIM,  if  additional  resources  are  needed  after  a  side  effect  has  been  performed, 
a  new  event  may  be  created  and  execution  of  the  current  event  brought  to  an 
end  This  new  event,  when  executed,  may  accumulate  such  resources  held  by  the 
old  event  as  are  still  needed,  plus  the  now  resources  desired,  and  then  proceed 
with  the  computation 

If  an  event  were  to  acquire  some  resources,  perform  a  side  effect,  and  then 
execute  forever,  no  opportunity  would  arise  for  aborting  it  and  freeing  thtr 
resources  it  had  tied  up,  and  hence  other  events  needing  those  resources  would 
be  prevented  from  ever  completing  If  execution  of  every  event  Is  guaranteed  to 
take  only  finite  time,  this  problem  can  be  avoided  (actually,  only  the  execution  time 
after  fbe  first  side  effect  need  be  finite),  hence  we  require  the  user  of  VIM  to  obey 
this  discipline  The  possibility  of  events  being  "starved  out”  from  access  to 
needed  resources  is  one  reason  for  insisting  that  execution  time  of  events  not  be 
Infinite  There  are  Other,  pragmatic  reasons  for  keeping  the  execution  time  for  any 
one  event  fairly  short  The  longer  the  execution  time  for  an  event,  the  more 
resources  it  is  likely  to  ne^d  to  acquire  This  increases  not  only  the  likelihood  that 
the  event  execution  will  be  aborted  before  it  is  able  to  complete  successfully,  but 
also  the  amount  of  work  that  is  lost  if  such  an  abortion  occurs  Thus  although  VIM 
is  a  perfectly  suitable  environment  for  running  lengthy  programs,  the  execution  of 
such  programs  should  tie  broke"  up  into  modest-length  event  executions.  In  the 
MuNet,  there  is  actually  a  bound  on  the  amount  of  time  the  execution  of  a  single 
event  is  allowed  to  take 

The  concept  of  event  abortion,  although  strictly  necessary  only  for  deadlock 
avoidance,  has  other  u  es  In  a  multiple-processor  system,  an  event  might  request 
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a  resource  inconvenient  to  supply  on  the  processor  where  the  event  is  currently 


being  executed  for  example,  an  event  might  request  access  to  an  object  text 
not  currently  present  on  that  processor,  or  ask  to  create  an  inconveniently  large 
object  In  such  a  case,  the  best  solution  may  well  be  to  transfer  the  event  to  a 
more  hospitable  location  It  will  frequently  be  more  economical  to  abort  tha  event 
execution  and  start  over  from  scratch  in  the  new  location  than  to  attempt  to 
transfer  all  the  intermediate  state  information  pertaining  to  the  event  execution 
Therefore,  under  VIM,  we  include  requests  to  allocate  storage  (create  events, 
create  or  expand  objects)  along  Aiith  requests  to  gain  access  to  object  texts  in 
the  class  of  operations  that  are  only  mgal  when  the  requesting  event  tias  not  yet 
performed  any  side  effects  Thus  on  event  execution  may  be  aborted  during  any 
such  request 

The  original  motivation  for  aborting  events  was  to  allow  resolution  of  deadlocks 
caused  by  requests  for  non-shared  access  to  an  object,  the*  concept  was  then 
found  useful  for  the  various  functions  described  in  the  preceding  paragraph  If 
only  shared  (read-only )  access  to  objects  is  allowed,  then  it  will  never  bo  neces¬ 
sary  to  abort  an  event  But  if  the  ability  to  obtain  read/ write  access  to  objects  is 
removed  from  VIM.  and  no  new  facility  put  in  its  place,  it  becomes  impossible  to 
perform  "joins"  or  any  kind  of  synchronization  between  parallel  processes  There 
do  exist  mechanisms  less  gnneiol  thin  the  unlimited  ability  to  perform  side  effects 
such  as  co/xh/its[ 38 ]  or  foAeos[16],  tha"  might  be  used  to  solve  this  problem 
without  introducing  the  need  to  he  able  to  abort  events.  VIM's  goal  of  being  able 
to  represent  most  general-purpose  computations,  however,  is  most  easily  served  by 
the  unencumbered  side-effect  mechanism  proposed  here 
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2.3:  Object  Types 

When  an  event  Is  to  be  executed,  the  machine  code  to  actually  be  interpreted 
by  VIM  is  determined  by  the  identity  of  ttie  event’s  target  object  Any  object  used 
as  a  target  object  must  contain,  within  its  text,  an  object  reference  designated  as 
that  target  object’s  type  In  the  blackboard  interpreter,  the  slot  containing  this 
reference  is  designated  by  the  reserved  symbol  t  The  text  of  the  type  object 
(whose  reference  is  found  in  this  slot)  is  interpreted  as  containing  the  machine 
code  to  be  executed  This  relationship  is  depicted  in  Figure  2  3  I  xecution  of  an 
event  begins  by  locating  the  text  of  the  target  object's  type  object,  and  transfer¬ 
ring  control  to  the  machine  code  found  therein  This  machine  code  may  contain 
ordinary  arithmetic  and  logical  manipulations,  along  with  requests  for  special  VIM 
services  such  ns  those  discussed  above 


I  vent 


T arget  object 


Type  object 


Figure  ?  3  Object  types 


The  reason  for  adding  the  extra  level  of  indirection  of  the  type  mechanism, 
rather  than  simply  finding  the  machine  code  to  be  executed  in  the  text  of  the  tar¬ 
get  object,  is  that  it  permits  easy  creation  and  manipulation  of  closure-like  objects 
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1  he  text  of  the  target  object  contains  the  execution  environment  (bindings  of  vari¬ 
ables  within  the  closure),  and  the  text  of  the  typo  object  contains  the  algorithm 
lhe  type  mechanism  thus  eliminates  the  need  to  copy  the  algorithm  into  every  clo¬ 
sure  In  terms  popularized  by  Hewitt,  the  type  object  contains  the  script  and  the 
target  object  the  rtcquaintancas[  1  8] 


2.4:  System  Services 

f  or  concreteness,  this  section  enumerates  the  fundamental  VIM  system  ser¬ 
vices  These  are  not  "operating  system"  services,  merely  the  most  basic  facilities 
provided  by  the  virtual  machine  itself 

gt«xt(ruf ). 

Obtains  access,  for  the  currently  executing  event,  to  the  text  of  the  object 
referenc  ed  by  ref.  in  read-only  mode  I  or  the  remaining  duration  of  the  event 
execution,  ref  may  be  used  in  specifying  words  or  references  to  be  read  out 
from  the  t«*xt  A  gtext  request  may  not  occur  after  a  side  effect  has  been  per¬ 
formed 

locktext(ref ), 

like  gtext  in  every  respec  t.  except  that  non  shared  (read/write)  access  to  the 
object  is  obtained 

newobj(.we)  returns  ref. 

Creates  a  new  object  with  a  text  of  the  specified  size  A  reference  to  the 
new  object  is  returned  in  ref  Mend/write  access  to  the  object  is  allowed  dur¬ 
ing  the  remainder  of  the  current  event  execution  A  newobj  call  <  az>  abort  the 
requesting  event  (this  allows  n  VIM  imjilementation  tr>  move  the  event  to  where 
more  sj>ace  is  available),  and  hence  may  not  be  made  after  a  side  effect  has 
been  performed 

n»wev(si.'e)  returns  evpo/nfer. 

Creates  a  now  event  of  the  size  requested,  and  returns  a  pointer  (which  may 
be  used  for  filling  in  the  event)  in  evpof.ofer  An  event  created  by  newev  will 
not  be  activated  if  execution  of  the  creating  event  does  not  complete  success¬ 
fully  Also  as  with  newobj.  a  newev  request  may  not  occur  following  a  side 
effect 
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done( ), 

Indicates  that  execution  of  the  current  event  tins  completed  All  objects  and 
events  created  by  it  are  officially  Installed,  and  the  current  event  is  deac¬ 
tivated 

This  set  of  primitives  provides  only  a  very  basic  and  unoptimi/ed  interface  to 
VIM  Calls  providing  additional  capabilities  are  discussed  in  subsequent  sections; 
modifications  to  VIM  to  Improve  the  efficiency  of  possible  Implementations  are 
touched  upon  In  Appendix  A 

2.5:  The  Blackboard  Interpreter 

The  blackboard  interpreter  for  VIM  has  two  components  a  graphical  represen¬ 
tation  for  the  state  of  a  computation,  and  a  set  of  transition  rules  specifying  what 
alterations  to  a  computation  state  will  yield  a  legal  successor  of  that  state  By 
starting  with  some  initial  state,  and  repeatedly  applying  the  transition  rules,  one 
can  generate  a  sequence  of  states  that  corresponds  to  a  legal  computation  in  VIM 
Due  to  the  fact  that  several  events  can  simultaneously  be  active,  as  well  as  the 
fact  that  it  is  possible  to  write  nondeterminate  programs  in  VIM.  a  state  will  in  gen¬ 
eral  have  several  legal  successor  states,  depending  on  the  focus  of  the  transition 
rule  applied  Indeed,  this  attribute  of  VIM  is  necessary  i*  we  are  to  imagine  VIM 
programs  executing  on  multiple  processors  Not  all  /ega/  successors  of  a  state  are 
equally  rfosirab/e.  though,  since  some  involve  regression  rather  than  progression  of 
the  computation  (aborting  an  event  is  an  example  of  a  regressive  step)  Some 
choices  of  successors  may  result  in  parts  of  a  computation  being  "starved  out"  or 
ignored  to  the  benefit  of  other  parts  of  the  computation  Avoidance  of  these  prob¬ 
lems  is  generally  associated  with  "fairness"  of  scheduling,  and  is  not  dealt  with  by 
the  rules  of  the  blackboard  interpreter  Thus  the  VIM  semantics  defined  by  the 
blackboard  interpreter  cannot  guarantee  tor  mi  nation  of  a  computation  They  can, 
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however,  assure  partial  correctness,  in  that  no  rule  allows  a  transition  which 


violates  any  ol  the  guarantees  which  VIM  makes  with  respect  to  the  meanings  of 
the  various  permissible  operations 


2.5.1:  Definition  of  the  Blackboard  Interpreter 

1  he  graphical  representation  of  objects  and  events  has  already  been  given  (in 
figures  P  1  and  P  P,  above)  One  additional  entity  appears  In  blackboard  interpre¬ 
tations  of  VIM  this  is  the  ai'tivat/on  record  (see  figure  P  4),  one  of  which  Is  asso¬ 
ciated  with  each  event  currently  being  executed  An  activation  record  for  an 
event  appears  when  the  event  begins  execution,  an  event's  activation  record 
disappears  when  it  either  successfully  completes  execution  or  Is  aborted  l very 
activation  record  is  drawn  just  to  the  right  of  the  event  it  corresponds  to  (an 
event  may  have  at  most  one  activation  record  at  one  time)  like  an  object  or 
event,  an  activation  record  has  several  slots,  each  bearing  a  symbolic  identifier  and 
containing  either  an  object  reference  or  some  kind  of  primitive  value  Slots  may  bo 
added  to  an  activation  record  as  execution  of  its  associated  event  proceeds 


event  activation  record 


Figu-e  2  4  An  activation  record  (with  associated  event) 


As  in  the  case  of  an  object  or  event,  special  slots  In  an  activation  record  may 
be  labeled  with  various  reserved  identifiers  In  fact,  every  activation  record  has 
several  such  slots  The  reserved  identifiers  used  in  activation  records  are  given  In 
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Identifier 


Slot  Contents 


t  a  reference  to  the  event's  target  object 

•  a  reference  to  the  event's  type  object 

•  the  current  program  counter  for  the  event 

•  a  Boolean  side  effect  flag 

af  an  access  privilege 

Table  ?.b:  Reserved  identifiers  used  in  activation  records 

Table  ?b  Several  new  terms  are  found  in  the  table.  The  program  counter  r 
names  the  slot  in  the  event’s  type  object  containing  the  next  machine  instruction 
to  be  executed  The  side  effect  flag  #  is  a  Boolean  which  is  initially  false,  but  is 
set  to  true  when  the  event  performs  its  first  side  effect  In  fact,  we  will  define  a 

"side  effect"  as  any  action  which  causes  the  side  effect  flag  to  be  set  to  true; 

the  specific  rules  governing  the  setting  of  the  side  effect  flag  will  be  given  below 
The  slots  contain  the  access  pr  Ivileges  obtained  thus  far  by  the  current  event 
(e.g..  via  gtext  or  locktext)  The  graphical  representation  of  an  access  privilege 
bears  a  superficial  resemblance  to  that  of  an  object  reference  both  are  drawn  as 
arrows  However,  to  denote  their  special  status,  the  arrowheads  associated  with 
access  privileges  are  drawn  with  serifs,  as  in  the  access  privilege  in  f  igure  ?  4 
An  access  privilege  that  has  been  granted  is  drawn  as  an  arrow  with  a  solid  shaft, 
a  privilege  which  has  been  requested  but  not  yet  granted  is  drawn  as  an  arrow 
with  a  dashed  line  for  a  shaft  (see  the  access  privilege  in  Figure  ?  4) 

To  distinguish  between  different  kinds  of  access  privileges,  arrows  with  multi¬ 
ple  heads  are  used  Table  2  6  summarizes  the  different  kinds  of  access  privileges 
and  the  monitor  calls  from  which  each  may  result 

Access  privileges  cannot  be  explicitly  handled  by  programs  in  VIM  In  particu¬ 
lar.  they  cannot  be  copied  out  of  the  slots  •)  Into  any  other  slots  In  an  activation 
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Access  Privilege 
> 

»» 

» 


Manning 

read-only  access  to  an  object  (gtext) 
read/write  access  to  an  object  (locktext) 
newly  created  object  (newobj) 
newly  created  event  Intwev) 
gtext  to  locktext  conversion  request 


Table  2  0  Access  privilege  types 


record,  nor  can  they  be  saved  in  event  or  object  slots  Access  privileges  exist 
only  to  record  permissions  accumulated  by  an  executing  event,  and  when  an 
event's  activation  record  disappears,  the  associated  access  privileges  must  disap¬ 
pear  also 

Before  giving  the  state  transition  rules  for  the  blackboard  interpreter,  it  is  con¬ 
venient  to  define  some  "canned  procedures"  as  follows 


IHMOH 

A  call  to  this  routine  indicates  that  an  erroneous  condition  exists  (/.e.,  the  user 
has  violated  some  rule  of  programming  in  VIV1  The  response  of  VIM  to  such 
error  conditions  is  not  specified  by  the  blackboard  interpreter 

MAS-GT L X T (event, obje<  t ) 

If  some  slot  a/  in  the  activation  record  for  event  has  an  access  privilege  (of  any 
type)  for  objei  t.  then  return  true,  else  return  false 

HAS-LOCK IT  X  1  (event, object ) 

If  some  slot  in  the  activation  record  for  event  has  a  locktext  or  newobj-type 
access  privilege  (serifed  double  or  triple  arrowhead)  for  object,  then  return 
true,  else  return  false 

GTC  XI  (event  ,ob  j  ect) 

If  MAS-GTF  XI (event, object)  then  return,  else  create  a  new  slot  in  the  activa¬ 
tion  record  for  event  and  place  In  it  a  gtext-type  access  request  (an  arrow  with 
a  dashed  shaft  and  a  serifed  single  arrowhead!  for  object 
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LOCK  1 1  XI (event, object) 

If  HAS-l  OCkTFXI(evenf,ob/*x:f)  then  return,  else  If  some  slot  af  in  the  activa¬ 
tion  record  for  event  has  a  gtext-type  access  privilege  (serifed  single  arrow¬ 
head)  for  object,  convert  that  access  privilege  to  a  gtext- to-locktext  conver¬ 
sion  request  (an  arrow  with  a  dashed  shaft  and  a  double  arrowhead  with  bent 
serifs)  for  object-,  else  create  a  new  slot  a/  in  the  activation  record  and  place 
In  it  a  locktext  type  access  request  (an  arrow  with  a  dashed  shaft  and  a  dou¬ 
ble  arrowhead  with  straight  serifs)  for  object. 

CRf  ATf  -ACT  IVA1  ION-RF  CORD(evonf ) 

Draw  a  blank  activation  record  to  the  right  of  event  If  event  has  no  slot 
labeled  r,  or  if  the  contents  of  slot  t  is  not  a  reference  to  an  object,  then 
ERROR;  else  copy  the  object  reference  from  slot  t  of  event  to  slot  r  of  the 
activation  record  Set  slot  #  of  the  activation  record  to  the  null  reference  Set 
slot  *  to  zero  and  slot  a  to  false  Perform  GTE  XT(eve/)f,  object  referenced  by 
slot  r  in  the  activation  record) 

F  RASF-AC I IVA  T  lON-RFCORD(evenf) 

f  rase  the  activation  record  to  the  right  of  event  Also  erase  all  object  refer¬ 
ence  and  access  privilege  arrows  (whether  their  shafts  are  solid  or  dashed 
lines)  emanating  from  the  erased  activation  record 

CAN-RUN(ovenf) 

If  no  access  requests  are  outstanding  in  the  activation  record  for  event  (r.e,,  no 
arrows  with  dashed  shafts  emanate  from  any  slot  in  the  activation  record), 
then  return  true;  else  return  false 

SIDI  -EF  F EC T-PERF ORMI  D(evenf ) 

If  slot  •  of  the  activation  record  for  event  contains  the  value  true,  then  return 
true,  else  return  false 


The  following  transition  rules  can  each  be  applied  to  any  event  e  meeting  the 
specified  conditions 


Rule  R1:  If  e  has  no  activation  record  and  the  active  flag  of  e  is  true,  then 
CRF  ATE-AC  T IVAI ION-FEE  OORD(e)  This  corresponds  to  beginning  execution  of  the 
event  e  and  gtextiny  the  event’s  target  object 

Rule  R2:  If  e  has  an  activation  record  and  SIDE-EF  FECT-PERFORMLD(e)  is  false, 
then  FRASE -ACTIVATION-RECORO(e)  This  corresponds  to  aborting  execution  of 
e. 
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Rule  R 3:  It  h  has  an  activation  record,  CAN-RUN(e)  is  true,  and  slot  #  of  the  activa¬ 
tion  record  contains  a  null  reference,  then  if  the  object  referenced  by  slot  r  in 
the  activation  record  has  no  slot  #  then  ERROR;  else  if  that  slot  •  does  not 
contain  a  reference  to  an  object  then  ERROR;  else  copy  that  reference  into 
slot  #  of  the  activation  record  for  e,  and  perform  CjTl  Xl(e,  object  referenced  by 
slot  t  in  the  activation  record  for  e).  This  corresponds  to  gtexting  the  event 
e’s  type  object. 

Rule  R4:  If  e  has  an  activation  record,  CAN-RUN(e)  is  true,  and  slot  t  of  the  activa¬ 
tion  record  does  not  contain  a  null  reference,  then  NT  XT-INSTRtJCTION(e)  The 
routine  NE.XT-INSTHUC  I  ION  is  described  later  in  this  section,  and  has  the  effect 
of  executing  the  next  Instruction  in  the  type  oliject  of  the  event  e 


The  following  transition  rules  can  be  applied  to  any  oliject  o  meeting  the 
specified  conditions. 


Rule  R5:  If  o  has  a  gtext-typi'  access  request  (dashed  arrow  with  serifed  single 
arrowhead)  impinging  on  it,  and  no  locktext-type  access  privileges  (solid  arrows 
with  serifed  double  arrowheads)  impinging,  then  convert  the  gtext-type  request 
into  a  gtext-type  access  privilege  (solid  arrow  with  serifed  single  arrowhead) 
This  corresponds  to  granting  shared  (read-only)  access  to  an  object  that  is  not 
currently  being  held  by  anyone  for  non-shared  (read/ write)  access 

Rule  R6:  It  o  has  a  locktext-typo  access  request  (dashed  arrow  with  straight- 
serifed  double  arrowhead)  impinging  on  it,  and  no  access  privileges  of  any  sort 
(solid  arrows  with  any  number  of  serifed  arrowheads)  or  gtext-to-locktext 
conversion  requests  (dashed  arrows  with  bent-s  rifed  double  arrowheads)  imp¬ 
inging.  then  convert  the  locktext-typo  request  into  a  locktext-type  access 
privilege  (solid  arrow  with  serifed  double  arrowhead)  This  corresponds  to 
granting  non-shared  access  for  an  object  not  currently  being  held  by  anyone  for 
any  kind  of  access 

Rule  R7:  If  o  has  a  gtcxt-to-locktext  conversion  request  (dashed  arrow  with 
bent-serifed  double  arrowhead)  impinging  on  it.  and  no  other  gtext-to-locktext 
conversion  requests,  or  access  privileges  of  any  sort,  impinginq,  then  convert 
the  gtext-to-locktext  conversion  request  into  a  locktext-typo  access  privilege 
(solid  arrow  with  serifed  double  arrowhead)  This  corresponds  to  upgrading  an 
event's  shared  access  to  an  object  to  non-shared  status 


This  completes  the  list  of  transition  rules  for  the  basic  VIM  blackboard  inter¬ 
preter,  except  that  the  routine  NEXT-INSTRUCTION,  which  is  referenced  in  the 
definition  of  Rule  R4  and  describes  the  interpretation  of  machine  code  found  in  type 
objects,  has  not  yet  been  described  The  purpose  of  postponing  this  until  last  has 
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been  to  abstract  from  the  presentation  the  details  of  any  particular  instruction  set 
for  VIM  The  goals  of  this  thesis  are  little  advanced  by  hypothesizing  in  detail,  for 
example,  the  permissible  set  of  addressing  modes  for  an  add  instruction  Conse¬ 
quently,  such  nuances  are  avoided  in  the  definition  of  Nl  XT-INSTRUCTION  Instead, 
the  body  of  the  definition  has  two  main  parts:  first,  a  section  dealing  with  the  VIM 
system  calls,  whose  format  may  be  considered  somewhat  standardized,  and  second, 
a  section  describing  the  treatment  of  different  kinds  of  accesses  to  objects  and 
events,  without  detailing  how  these  basic  kinds  of  accesses  are  packaged  Into 
Instructions  The  definition  of  NL  XT-INSTHUCTION  folio  vs 


NFXT-INSTRUCTION(  event): 

fetch  the  contents  of  the  slot  in  event's  type  object  (referenced  by  slot  $  in 
the  activation  record  of  event)  designated  by  the  contents  of  slot  *  (the  pro¬ 
gram  counter)  In  the  activation  record  for  event  Interpret  the  datum  thus 
fetched  as  an  instruction  If  the  Instruction  is  a  VIM  system  call,  then  it  is  one 
of  the  following 

done()  for  every  In  the  activation  record  for  event  that  contains  a 
newev-type  access  privilege  (arrow  with  serifed  quadruple  arrowhead), 
set  to  true  the  active  flag  of  the  event  pointed  to  by  the  access 
privilege  Set  to  false  the  active  flag  of  event  FRASF-ACTIVA7ION- 
Rf  CORO(evenf) 

gtext(ob/ecf)  *  If  SIDF-Ff  FFCT-Pf  RFORMf  D( event)  then  FflROR;  else 
advance  *  to  the  next  Instruction  and  perform  GTE XT(event, object). 

locktext (ob/ecfi  If  SIDF-FFFFC  T-PF  Rf  ORMf  0(event )  then  ERROR;  else 

advance  w  to  the  next  instruction  and  perform  IOCK.TF  XT(evenf,objecf). 


"Although  we  have  not  specified  the  means  by  which  the  reference  ob/ecf  is 
obtained,  in  this  and  all  subsequent  cases  it  is  assumed  that  object  came  eithei 
from  event,  from  the  activation  record  for  event,  or  from  the  text  of  some  object  for 
which  at  least  gtext-type  access  has  already  been  established  by  event  The 
Intent  of  this  stipulation  is  to  ensure  that  all  references  used  by  an  event  are 
accessible  from  It  via  some  chain  of  object  references. 
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n*\Mob}(.id^,i/aly.ld2,i/al2.  ..,ian,valn).  If  SIDL-L IHC1  -PLREOHMF  D(eve/>f )  then 
FRROR,  else  advance  w  to  the  next  instruction,  draw  a  new  object  with 
n  slots  labeled  id  yid 2,...,idn,  with  initial  contents  val  yval  2,...,val  n,  create 
a  new  slot  in  the  activation  record  for  event,  and  draw  a  newobj-type 
access  privilege  (solid  arrow  with  sented  triple  arrowhead)  from  this  slot 
to  the  newly  created  object  The  newobj  operation  returns  a  reference 
to  the  newly  created  object. 

nevjen(idy.valy,id2.val?,...,idn,valn):  If  SIDL-EEFLCl-PERFORMl  D(evenf)  then 
FRROR;  else  advance  *  to  the  next  instruction,  draw  a  new  event  with 

n  slots  labeled  id  ..id  2 . id  r>,  with  initial  contents  val  yvalp,.  ,valn,  create 

a  new  slot  a/  in  tne  activation  record  for  event,  and  draw  a  newev  lype 
access  privilege  (solid  arrow  with  serifed  quadruple  arrowhead)  from  this 
slot  to  the  newly  created  event,  and  set  the  active  flag  of  the  newly 
created  event  to  false  The  newev  operation  returns  an  integer  which 
may  be  used  to  refer  to  the  now  event 

If  the  instruction  is  not  a  VIM  system  call,  then  it  may  attempt  some  combination 
of  the  following  operations. 

read  from  activation  record  for  event  if  reading  from  one  of  the  slots  o^, 
then  FRROR,  else  proceed 

write  into  activation  record  for  event  if  writing  slot  e  or  any  of  the  slots  a f, 
then  FRROR,  else  proceed 

read  from  event  proceed 

write  into  evenr  FRROR 

read  from  an  event  other  than  event  f  RROR 

write  into  an  event  other  than  event  if  a  newev-type  access  privilege  for 
the  destination  event  exists  in  some  slot  o(  in  the  activation  record  for 
event,  then  proceed,  else  ERROR 

read  from  text  of  ob/ecf  if  HAS-GTf  XT(evenf,ob/ecf)  then  proceed;  else 
FRROR 

write  into  text  of  object  if  a  newobj-type  access  privilege  for  object 
exists  in  some  slot  a(  in  the  activation  record  for  event,  then  proceed, 
else  if  a  locktext-type  access  privilege  for  object  exists  in  some  slot  o( 
in  that  activation  record,  set  slot  *  in  the  activation  record  to  true,  then 
proceed,  else  FRROR 


One  additional  important  requirement  of  the  blackboard  interpreter  has  not  yet 
been  stated  in  this  section:  this  Is  the  requirement  that  the  execution  of  any 
event  take  only  a  finite  number  of  s'.<  ps  after  the  first  step  that  causes  a  side 
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effect  (i.e.,  causes  slot  #  of  the  activation  record  to  be  set  to  true).  Ihis  require¬ 
ment  may  be  imposed  on  the  user  In  one  of  two  ways.  It  may  simply  be  stated 
that  any  program  which  might  execute  forever  after  performing  its  first  side  effect 
Is  illegal,  without  any  indication  of  how  such  an  illegal  program  could  be  detected 
I  his  would  impose  on  the  user  the  onus  of  ensuring  the  finiteness  of  every  event 
execution  once  it  has  performed  a  side  effect.  Alternatively,  some  number  of 
steps,  such  as  10,000,  could  be  picked  arbitrarily  as  the  maximum  number  of  steps 
permissible  after  performing  a  side  effect  Any  event  execution  exceeding  this 
limit  could  be  terminated  with  an  error  condition,  much  as  it  would  be  if  it  attempted 
to  obtain  additional  access  privileges  after  performing  a  side  effect 

2.6.2:  Discussion  of  the  Blackboard  Interpreter 

Before  discussing  the  interaction  of  the  various  parts  of  the  blackboard  inter¬ 
preter,  it  is  best  to  illustrate  the  interpreter's  operation  by  moans  of  an  example 
In  order  to  give  an  example,  it  is  necessary  to  settle,  at  least  informally,  on  some 
sample  "machine  language"  with  which  to  populate  the  slots  of  typo  objects  At 
the  foundation  of  this  language  must  be  some  notation  for  operand  locations  we 
must  be  able  to  name  the  slots  from  which  operands  are  to  be  fetched  or  into 
which  results  are  to  be  put  f-or  this  purpose  we  settle  on  the  following  conven¬ 
tion  a  lone  identifier  *  denotes  the  slot  (or  its  contents,  depending  on  whether 
the  identifier  appears  as  a  destination  or  as  a  source)  bearing  the  label  x  in  the 
currently  executing  event's  activation  record  A  pair  x:y  denotes  the  slot  labeled  y 
in  the  object  referenced  by  the  contents  of  slot  *  in  the  activation  record  A  pair 
«:y  denotes  the  contents  of  slot  y  in  the  currently  executing  event. 

Slots  are  stored  into  by  using  the  form  /ocn*  va/i/e,  where  ira/r/e  may  denote 
the  contents  of  some  slot,  as  discussed  above,  or  may  be  the  result  returned  by 
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some  system  call  such  as  newobj.  locn  will  denote  the  slot  into  which  the  value 


should  be  stored  If  locn  is  a  lone  identifier,  then  a  new  slot  by  that  name  will  be 
added  to  the  current  activation  record,  if  no  such  slot  existed  before,  otherwise, 
the  slot  named  by  locn  must  already  exist 
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figure  2  /'(a)  Initial  configuration 


This  "machine  language"  suffices  to  present  simple  examples  of  blackboard 
Interpretations  1  ho  reader  will  indulge  one  additional  piece  of  novelty  drawing 
type  objects  vertically  rather  than  horizontally  to  more  easily  accommodate  the 
symbolic  representations  of  the  instructions  in  their  slots.  Consider  then  the  black¬ 
board  interpretation  shown  in  figure  2  7  Along  with  each  part  of  the  figure  is 
listed  the  blackboard  interpreter  rule  whose  application  produces  the  configuration 
shown  in  that  part  from  the  configuration  in  the  previous  part 

figure  2.7(a)  shows  the  initial  configuration,  which  includes  one  event  (with  its 
active  flag  set  to  true)  and  a  collection  of  objects  The  event's  type  object, 
drawn  vertically,  contains  the  program  to  be  executed  The  slots  In  this  object  are 
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labeled  with  numbers,  corresponding  to  values  that  will  be  taken  on  by  the  program 
counter  during  execution. 
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Figure  ?  /(b)  Produced  from  ?  7(a)  by  Rule  R1. 
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figure  2.?(d)  Produced  from  2.7{ c)  by  Rule  R3 

Figures  2  7{ b)  through  (e)  show  the  standard  start-up  sequence  for  events; 
note  that  no  step  in  this  sequence  depends  on  the  program  contained  in  the  type 
object  This  event  start  up  phase  thus  corresponds  to  the  noxt-mstruction-fetch 
phase  typical  of  standard  von  Neumann  machines,  which  is  the  same,  ip  to  some 
point,  no  matter  what  instruction  is  being  fetched  (since  the  nature  of  that  instruc¬ 
tion  is  not  known  yet1)  In  our  blackboard  interpreter,  this  "fetch"  phase  collects 
references  and  read-only  access  privileges  for  the  target  and  type  objects  of  the 
event 
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Figure  ?  7(g)  Produced  from  P.7(f)  by  Rule  R4 


Figure  2  7(h)  Produced  from  2  7(g)  by  Rule  R4 
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Figure  2.7(1)  Produced  from  2  7(h)  by  Rule  Rf> 

Actual  execution  of  the  machine  code  in  figure  2  7  begins  with  the  transition 
between  snapshots  (e)  and  (f)  Starting  at  this  point,  every  application  of  Rule  H4 
corresponds  to  the  execution  of  another  Instruction  In  snapshots  (f)  and  (g),  the 
activation  record  is  extended  with  new  slots  a  and  b.  and  initial  contents  for  those 
slots  are  supplied  the  requesting  and  granting  of  gtext  and  locktaxt  privileges 
occupies  snapshots  (h)  through  (k)  Snapshot  (I)  shows  the  result  of  the  first  (and. 
In  this  case,  only)  "side  effect"  performed  during  ex»»cution  of  the  event  the 
reference  from  the  slot  »:x  has  been  copied  into  skit  b:l  and  the  side  effect  flag  * 
has  been  set  to  true  Note  that  gtext  access  has  been  obtained  for  object  a  and 
locktaxt  access  for  object  b.  otherwise  the  operation  "b:I  •  a:l"  would  bo  illegal 
Finally,  snapshot  (m)  shows  the  result  of  executing  the  dona  system  call  the 
activation  record  and  all  arrows  emanating  from  It  have  been  erased,  and  the 
active  flag  of  the  event  has  been  set  to  false,  ensuring  that  its  execution  will  not 
be  attempted  again 
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Figure  2  7(j)  Produced  from  2  T( i)  by  Pule  R4 
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Figure  2./(k)  Produced  from  2 V(j)  by  Rule  R6 
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Figure  2  7( I):  Produced  from  2.7( k)  by  Rule  R4 


Several  aspects  of  the  sequonce  in  Figure  2.7(a)  through  (m)  are  worth  noting 
The  first  is  that,  since  only  one  evont  execution  was  involved,  there  was  no  oppor¬ 
tunity  for  parallelism  In  fact,  at  any  given  point,  there  was  only  one  rule  that 
could  be  applied,  except  that  at  any  point  before  Figure  27(1),  Rule  R?  could  have 
been  applied,  aborting  the  event  execution  and  restoring  the  state  In  figure  2.7(a) 
The  property  illustrated  by  this  is  true  in  general  any  time  an  event  is  aborted.  the 
resulting  state  is  indistinguishable  from  th*>  state  that  would  have  resulted  if  the 
at'orte<i  event  «•••  ut  on  had  never  been  attemptf'd  The  VIM  rules  regulating  the 
setting  of  the  side-effect  flag  *,  in  combination  with  the  restriction  that  an  event 
can  only  be  abc  ted  when  the  side-effect  flag  t  in  its  activation  record  Is  false, 
ensure  that  this  is  the  case 
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figure  2  7(m)  Produced  from  2.7(1)  by  Rule  R4 


More  specifically,  we  can  define  an  access/6/e  event  as  being  any  active  event, 
and  an  access/b/e  obje<  t  as  being  any  object  reachable  by  some  chain  ot  object 
references  from  some  ac'ive  event  or  its  activation  record  When  an  event  exe¬ 
cution  is  aborted,  it  may  have  created  some  events  or  objects,  which  will  be  left 
around  afterwards,  but  these  events  and  objects  will  always  be  inaccessible  The 
newly  created  events  will  be  inaccessible  because  their  active  flags  will  still  be 
set  to  false  (very  newly  created  object  will  be  inaccessible  because,  in  order  for 
It  to  become  accessible,  a  reference  to  it  would  have  to  have  been  stored  into  the 
text  of  some  previously  accessible  object  Any  event  performing  such  an  opera¬ 
tion,  however,  would  have  its  side-effect  flag  *  sot  to  true,  rendering  subsequent 
abortion  of  the  event  impossible  Thus  if  we  consider  only  the  states  of  accessible 
events  and  objects  to  be  relevant  (a  reasonable  assumption,  since  the  others  can 
never  Influence  any  future  computation),  an  aborted  event  execution  leaves  behind 
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the  same  result  as  an  event  execution  that  never  happened 

An  Interesting  blackboard-interpreter  feature  Illustrated  by  figure  2.7  is  the 
way  in  which  control  is  passed  back  and  forth  between  events  and  objects  In 
general,  control  resides  with  events.  In  the  sense  that  transition  rules  are  applied 
at  events,  and  primarily  Involve  the  contents  of  events,  activation  records,  and 
their  associated  target  and  type  objects  However,  any  time  an  event  makes  a 
gtext  or  tocktext  request  to  access  an  object,  no  further  rules  (other  than  Hule 
H 2,  aborting  it)  can  be  applied  to  the  event  until  the  requested  access  is  granted 
Granting  this  access  is  under  control  of  the  object,  in  the  sense  that  the  relevant 
rule  (Hfj.  R6,  or  Hz')  operates  in  the  vicinity  of  the  object,  taking  into  consideration 
only  the  other  access  privileges  already  granted  for  that  object  The  form  of  arbi¬ 
tration  among  access  requests  specified  by  these  rules  is  what  guarantees  that 
gtext  access  will  only  be  shared  with  other  readers,  and  that  locktext  access  will 
bo  shared  by  no  one 

The  nature  of  these  arbitration  rules  in  turn  guarantees  a  second  fundamental 
property  of  VIM  If  we  define  an  execution  history  as  a  series  of  snapshots,  such 
as  those  in  figure  2.7,  then  for  any  legal  VIM  execution  history  there  exists  a  his¬ 
tory  starting  with  the  same  initial  state  and  ending  with  tht;  same  final  state,  which 
has  the  property  that  at  most  one  activation  record  exists  in  any  one  snapshot  In 
other  words,  any  arbitrarily  parallel  execution  of  some  set  of  events  cannot  yield  a 
result  that  could  not  also  be  yielded  by  some  sequential  execution  in  which  no  two 
event  executions  are  over  interleaved  This  property  of  VIM  simplifies  The 
programmer's  job,  for  he  can  treat  each  event  execution  as  though  it  were  an 
atonic  operation  which  cannot  be  interrupted  by  any  complete  or  partial  execution 
of  another  event 

Every  VIM  execution  history  can  be  viewed  as  a  sequence  of  primitive 
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operations,  corresponding  to  the  successive  applications  of  the  blackboard- 
interpreter  rules  to  various  events  and  objects  I  ach  application  of  a  rule  can  be 
associated  with  the  execution  of  a  particular  event  In  the  case  of  a  rule  that 
applies  directly  to  an  event,  this  event  is  the  event  associated  with  the  primitive 
operations  performed  In  the  case  of  a  rule  that  is  applied  at  an  object,  fo  grant 
an  access  privilege,  the  event  to  which  access  is  granted  is  the  event  associated 
with  the  primitive  operation 

s 

In  a  VIM  execution  history  in  which  events  are  executed  in  sequence,  the 
primitive  operations  corresponding  to  each  event  execution  will  be  grouped 
together,  with  no  interleaving  of  primitive  operations  belonging  to  different  event 
executions  In  a  general  VIM  execution  history,  this  will  not  be  the  case;  how¬ 
ever.  any  logal  V//V f  execution  history  can  bo  tr  ansi  or  mod  into  ono  in  which  primi- 
tivo  oporations  portaming  to  tiAch  event  execution  aro  gr  on  pod  togothor,  without 
altering  the  initial  or  final  state  1  he  transformation  is  simple  to  perform  without 
any  change  in  the  ordering,  relative  to  each  other,  of  the  primitive  operations  per¬ 
taining  to  each  event  execution,  a  new  history  is  built  out  of  these  sets  of  primi¬ 
tive  operations,  in  which  the  sets  appear  in  the  order  in  which  the  corresponding 
event  executions  in  the  old  history  ended  (i.e.,  either  completed  successfully  or 
were  aborted)  In  other  words,  all  primitive  operations  in  the  original  history  are 
delayed  as  long  ns  possible,  until  they  run  into  the  primitive  operation  representing 
the  termination  of  the  corresponding  event  execution  (this  operation  should  not  be 
delayed)  This  strategy  will  produce  the  appearance  that  all  operations  associated 
with  an  event  execution  happen  in  a  burst,  immediately  before  the  termination  of 
that  event  execution 

Obviously  this  transformation  can  be  made  on  any  execution  history,  but  the 
contention  that  the  transformation  is  legal,  r.e.,  does  not  affect  the  rtate  change 
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performed  by  the  execution  history,  needs  to  be  justified  A  sufficient  condition  for 
the  transformation  being  valid  is  the  following  given  an  event  execution  B  that  ter¬ 
minates  alter  another  event  execution  A.  and  a  primitive  operation  pertaining  to  B 
that  occurs  immediately  before  a  primitive  operation  pertaining  to  A,  the  order  of 
the  primitive  operations  may  be  switched  without  altering  the  stale  change  per¬ 
former 1  by  the  pair  of  primitive  ofierations  This  property,  if  true,  can  be  used  to 
|ustlfy  "percolating"  operations  pertaining  to  event  executions  with  earlier  termina¬ 
tion  times  forward  and  those  corresponding  to  later  termination  times  back,  until  no 
operation  appears  before  another  one  whose  event  execution  ended  later 
precisely  the  situation  we  are  trying  to  achieve. 

In  trying  to  establish  this  property,  we  must  concentrate  on  the  ways  in  which 
event  executions  can  interact  All  such  ways  Involve  reading  or  writing  of  object 
texts,  or  manipulation  of  access  privileges  to  objects  Operations  involving  an 
event’s  activation  record  are  of  no  interest,  since  the  activation  record  can  only 
be  accessed  by  the  event  it  belongs  to  Similarly,  operations  that  involve  reading 
or  writing  events  are  of  no  interest  an  event  can  only  be  written  by  its  creator, 
during  which  time  no  other  event  execution  can  access  it,  and  it  can  only  be  read 
during  its  own  execution,  which  cannot  overlap  with  its  creation.  Thus  only  manipu¬ 
lations  involving  objects  can  load  to  interference  between  event  executions. 

The  primitive  operations  that  affect  objects  can  be  grouped  into  five  classes 
reading  from  an  object,  writing  to  an  object,  requesting  access  to  an  object,  being 
granted  access  to  an  object  (this  includes  gtaxt  access,  locktext  access,  and 
object  creation  which  entails  a  grant  of  nawobj-type  access  to  the  newly  created 
object),  and  relinquishing  access  to  an  object  (this  only  happens  when  an  event 
execution  ends).  The  only  way  events  can  conflict  is  on  operations  involving  the 
same  object,  clearly,  a  pair  of  primitive  operations  on  different  objects  miy  be 
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performed  in  either  order  table  2.H  shows  the  status  of  the  various  combinations 


of  possibly  conflicting  operations  on  the  same  object. 


_ 

A  read 

A  write 

A  request 

1  A  a 

A  grant 

A  relinquish 

B  read 

OK(  1 ) 

NP(2) 

0K(4) 

f  GT(6) 

OK(5) 

B  write 

NP(2) 

NP<2) 

OK(4) 

NP(2) 

OK(5) 

B  request 

OK(4) 

OK(4) 

OK(4) 

j  OK(4) 

OK(4) 

B  grant 

GT  (6) 

NP(2) 

OK(4) 

GT(6) 

OK(5) 

B  relinquish 

NP(3) 

NP(3) 

NP(3) 

1  NP(3) 

NP(3) 

(vent  execution  B  is  Assumed  to  have  ended  otter  event  execution  A;  each 
entry  in  the  table  applies  to  a  pair  of  primitive  operations  on  the  same 
object  in  which  the  operation  pertaining  to  B  occurs  immediately  brdore  that 
pertaining  to  A  4  key  to  the  table  entries  is 

NP:  this  sequence  of  operations  is  not  possible  in  a  legal  VIM  execution 
history. 

OK:  this  pair  of  oper  at  ions  performed  in  the  reverse  order  will  still  cause 
the  same  state  change. 

GT;  if  this  pair  of  operations  is  part  of  a  legal  VIM  execution  history, 
then  all  accesses  granted  must  be  of  gtext  access;  In  this  case,  per¬ 
forming  the  operations  in  the  reverse  order  will  still  cause  the  same 
state  change. 

Par enthesi sed  numbers  following  the  ttible  entries  refer  to  more  detailed 
explanations  in  the  ten' 

Table  ?H  Legality  of  primitive  operation  sequence  changes 


Explanations  associated  with  parenthetical  numbers  in  the  table  entries  are  as  fol 
lows 


(1)  Certainly  interchanging  the  order  of  two  reads  will  not  change  the  data  read 
by  either 

(2)  If  either  A  or  B  has  access  to  write  the  object,  the  other  cannot  possibly 
have  or  be  granted  access  to  read  or  write  it  until  the  first  access  privilege 
Is  relinquished 
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(3)  Since?  access  privileges  are  only  relinquished  when  an  event  execution 
ends,  and  event  execution  A  is  assumed  to  have  completed  before  the  end 
of  event  execution  B,  B  cannot  possibly  relinquish  any  access  privileges 
prior  to  any  operation  pertaining  to  A 

(4)  The  existence,  or  absence,  or  requests  (as  opposed  to  granted  privileges) 
from  an  event  does  not  interfere  with  any  operation  performed  by  another 
event  Thus  the  ordering  of  a  request  by  one  event  and  any  operation  by 
another  event  may  be  freely  interchanged 

(5)  If  a  read,  write,  or  grant  pertaimnc  to  B  were  legal  before  A  s  relinquishment 
of  access  privileges,  it  will  certainly  be  legal  after 

(6)  Both  A  and  B  have  access  to  the  object  by  the  end  of  the  pair  of  opera¬ 
tions  The  only  way  this  can  happen  is  if  both  have  gtext  access,  other¬ 
wise,  sharing  would  be  prohibited  If  the  access  grants  are  all  of  gtext 
access,  then  the  order  of  the  operations  is  irrelevant. 

In  summary,  then,  a  VIM  execution  history  may  be  viewed  as  a  sequence  of 
primitive  operations  Any  legal  VIM  execution  history  can  be  transformed  into  an 
equivalent  one  (performing  the  same  state  change)  in  which  event  executions  hap¬ 
pen  in  sequence,  in  the  order  in  which  the  event  executions  terminate  in  the  origi¬ 
nal  execution  history  This  transformation  can  be  performed  incrementally  by  a 
"bubble  sort"  techn.que,  interchanging  the  order  of  pairs  of  adjacent  primitive 
operations  where  their  order  is  the  opposite  of  that  desired  in  the  final,  sequential 
execution  history  The  considerations  presented  in  Table  PS  show  that,  if  the  ori¬ 
ginal  execution  history  is  a  legal  one.  such  order  changes  will  always  be  legal, 
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even  when  they  apply  to  pairs  ot  operations  on  the  same  object 

In  the  scenario  of  Figure  2.1 ,  there  was  always  one  transformation,  other  than 
aborting  execution  of  the  event,  that  could  be  applied  In  the  presence  of  several 
events,  there  might  at  any  point  be  several  possible  transformations:  on  the  other 
hand,  there  might  be  none,  as  illustrated  in  the  snapshot  of  f  igure  2.9. 
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Figure  2  9  Deadlock  in  locktexts 


This  snapshot,  from  which  inessential  details  have  been  omitted,  shows  a  deadlock 
situation  Ne.thor  event  can  continue  execution  because  each  is  waiting  for  a 
request  to  be  granted,  and  neither  request  can  be  granted  because  each  conflicts 
with  another  request  that  has  already  been  granted  (hero  is  a  case  where  the 
arbitration  function  of  objects  substantively  affects  the  course  of  execution)* 
Thus  the  only  choices  available  are  to  abort  one  or  both  events  Aborting  an  event 
will  erase  some  previously  granted  access  privileges,  removing  the  obstacles  to 


"Situations  similar  to  this  conflict  between  locktext  requests  can  arise  out  of 
conflicts  between  gte  xt  and  locktext  lequests 
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granting  other  requests  Since  an  event  is  forbidden  to  make  gtext  and  locktext 
requests  after  performing  a  side  effect  (/.e„  after  performing  any  operation  that 
sets  the  side-effect  flag  a  to  true),  any  event  with  pending  requests  that  have  not 
yet  been  granted  can  always  be  aborted  Conversely,  any  event  that  has  per¬ 
formed  a  side  effect  (and  therefore  can  no  longer  be  aborted)  cannot  be  prevented 
from  executing  because  of  unsatisfied  access  requests  Given  our  earlier  assump 
tion  that  the  execution  time,  or  number  of  Instructions  executed,  for  any  one  event 
is  finite,  the  third  fundamental  guarantee  about  execution  of  VIM  programs  follows 
no  um  esolvuble  (load lock  con  occur  If  an  event  holds  access  privileges  that  are 
preventing  execution  of  another  event  from  proceeding,  and  the  former  event  has 
not  yet  performed  a  side  effect,  the  access  privileges  it  holds  may  always  be 
released  by  aborting  it  If  the  event  holding  access  privileges  has  performed  a 
side  effect,  after  some  finite  number  of  steps  it  will  have  finished  executing,  at 
which  point  its  access  privileges  will  be  released 

Inspection  of  the  rules  of  the  blackboard  interpreter  yields  a  fourth  theorem 
no  event  can  conflict  with  itself  (i.e.,  cause  a  "deadlock"  with  itself  because  of  an 
unfortunate  sequence  of  access  requests)  At  first  glance,  it  might  seem  that  this 
could  happen  if,  for  example,  the  same  event  issued  two  locktext  requests  for  the 
same  object  The  blackboard-interpreter  routines  GTCXT  and  10CK.TFXT,  however, 
always  check  if  the  requested  access  has  already  been  obtained  by  the  current 
event,  and  thus  avoid  making  any  such  redundant  access  requests  Therefore,  if 
not  interfered  with,  any  event  can  eventually  finish  execution 

The  various  guarantees  made  by  the  rules  of  execution  of  VIM  programs  may 
be  summarized  as  follows: 
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•  since  the  effects  of  aborted  event  executions  cannot  be  detected,  the  pro¬ 


grammer  need  not  concern  himself  with  the  possibility  of  that  such  execu- 

/' 

(tions  could  happen 

•  since  all  deadlocks  at  the  g»  ext  locktext  level  can  be  handled  automatically, 
the  programmer  need  not  concern  himself  with  the*  possibility  of  such 
deadlocks 

•  since  no  rosult  can  bo  obtained  which  could  not  result  from  some  sequential 
execution  of  events,  the  programmer  need  not  concern  himself  with  the  pos¬ 
sibility  that  other  events  might  be  executing  concurrently  with  the  execu¬ 
tion  of  his  Of  course,  the  programmer  stilt  has  to  worry  about  the  conse¬ 
quences  of  other  event  executions  intervening  between  executions  of  any 
two  consecutive  events  of  his 


2.5.3:  An  Extended  Blackboard  Interpreter  Example 

As  a  more  realistic  example  of  the  user  of  the  blackboard  interpreter,  we  now 
consider  the  sequence  of  snapshots  in  Figures  ?  10(a)  through  (k)  The  program 


shown  in  those  snapshots  computes  the  nth  Fibonacci  number  f  using  the  relation 


/I  if  n  <  ? 

f”  "  lfn  2*fn  1  ,f  n  2 


(2.1  ) 


If  n  2  2,  the  algorithm  spawns  parallel  computations  of  f n  ^  and  ^  In  the  par 
tlcular  case  of  Figure  2.10,  the  program  is  used  to  compute  f ^ 
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Figure  2  10(n)  Initial  configuration  for  Fibonacci  example 


In  figure  ?  10(a).  the  sole  event  names  the  Fibonacci  actor  as  its  target 
object,  gives  the  value  3  for  the  argument  n.  and  contains  in  slot  c  a  reference  to 
a  continuation  object  to  'ecetve  the  result  Ttie  contents  of  the  continuation 
object  are  not  shown  here  The  target  object  in  figure  ?  10(a)  contains  only  one 
reference,  to  its  tyoe  object  The  typo  object  contains  in  slots  0  through  6  part 
of  the  f  Ibonacct  program,  and  in  slot  c  a  rofeience  to  another  object  containing  the 
remainder  of  the  program  This  arrangement  illustrates  a  general  property  of  type 
objects  na.  iely  that  some  slots  may  contain  data  oven  while  other  contain  instruc¬ 
tions  The  program  is  best  understood  by  watching  its  execution  unfold,  hence  its 
logic  Is  not  expialr  ed  h«  re  The  "machine  language"  used  In  It  has  a  few  new 
features,  though  first  of  all,  arithmetic  expressions,  rather  than  just  slot 
Identifiers,  are  used  on  the  right-hand  sides  of  assignment  statements  and  as 
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parameters  to  system  calls.  Secondly,  the  form 


If  e*press/on  goto  n 


is  used  to  cause  the  program  counter  w  to  be  set  to  n  If  e» pression  Is  true  This 
provides  a  primitive  facility  for  conditional  branching 


k1  fn]  fcT7>>T-  •'  P  P  $  r* '  r-  P  P 

L  f  1  3 :  f  *  4  *  -L*  4-'  4-l-fl-4i-f-i- 


)  Wr  _} 

C~ 


\A  V  « 

V  P  n  To  e 


'  '  I — F — 1  ” 

\t  n  c 

f  ■  i1 

L  *  .  . 1  1  *  L 


TmKrST^ 


0  If  t:n<2  goto  5 

1  X'  n«wobjlr.i:c.f  f:r.«.0) 

I  |  — 

2  n«wev(r.i-f.ni:n  2.c  x) 

3  n«wBv(r.i:r,n.i:n  l.c.x) 


4  don#( ) 

k  » - 1 

5  ncwev(r  «:c.f  1  ) 


11  > 

>0  if  r:a<0  goto  4 

1  locktext(r) 

2  r  : *•  «  :f 

<  *  { 

3  done! ) 


4  nBwev(r,r:c.f.r:»n:f) 


6  done() 


j  |5jdon«( ) 


F'gure  2  10(b)  Continuation  of  I  ibonacci  example 


Figure  P  10(b)  shows  the  state  of  affairs  after  execution  has  proceeded  past 
slots  0.  1  2.  and  3  of  the  type  object  The  figure  shows  the  new  object  and  two 
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now  events  created,  and  the  activation  record  entries  and  access  privileges 


created  In  the  process  Note  that  no  operation  performed  during  this  event  execu¬ 
tion  caused  the  side-effect  flag  *  to  be  set  to  true  lo  keep  the  diagram  from 
becoming  even  more  cluttered,  the  convention  has  been  established  that  drawing 
an  arrow  to  the  shaft  of  another  arrow  pointing  at  an  object  Is  as  good  as  drawing 
an  arrow  directly  to  the  object  itself  1  his  applies  both  to  regular  object  reference 
arrows  and  to  special  access  privilege  arrows 


figure  2  10(c).  Continuation  of  Fibonacci  example 
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In  figure  2.10(c),  execution  of  the  original  event  has  finished,  as  can  be  seen 
from  the  tact  that  its  activation  record  has  been  erased  and  its  active  flag  set  to 
false  Upon  successful  completion  of  its  execution,  the  two  events  created  by  it 
were  activated  and  they  now  a  wait  execution  Thes<<  events  represent  recursive 
calls  to  the  f  ibonacci  program  to  compute  f  ^  and  f  ,,  whose  sum  will  be  the  desired 
answer  f ,,  The  target  object  for  these  events  is,  of  course,  still  the  Fibonacci 
actor  Both  events  give  as  their  continuation  a  reference  to  a  newly  created 
object  whose  function  will  he  to  collect  the  two  results  generated  from  the  compu¬ 
tations  of  f.  and  f  ,,  add  them,  and  forward  the  sum  to  the  original  continuation 

I  t 

supplied  in  the  request  to  compute  f  ^ 
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figure  ?  10(d)  Continuation  of  Fibonacci  oxomplo 


For  the  transition  to  figure  ?  10(d),  wo  have  arbitrarily  chosen  to  execute  to 
completion  the  event  whose  job  was  to  compute  t ^  As  before,  this  has  led  to  the 
creation  of  two  new  events  (for  computing  and  f^)  and  a  new  object  This  new 
object  is  named  as  the  continuation  of  each  of  the  two  new  events,  and  will  func¬ 
tion  to  collect  the  values  returned  by  them  and  return  their  sum  as  the  value  of  1  ^ 
The  fact  that  recursive  calls  to  the  Fibonacci  actor  are  occurring  is  evidenced  by 
the  stack-like  structure  of  continuations  that  is  developing  Since  each  call  to  the 
actor  generates  two  new  calls  if  n  2  2,  the  stack-like  structure  is  really,  in  general. 
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a  tree  like  structure,  however,  this  example  Is  too  small  to  demonstrate  that 

the  existence  of  these  multiple  continuations  also  illustrates  the  motivation 
behind  the  target-object/type-ohject  mechanism  t  ach  continuation  has  a  slot  a 
which  records  the  state  of  the  particular  computations  that  are  to  return  values  to 
that  continuation  If  the  slot  a  contains  *ero,  both  computations  are  still  active  If 
the  contents  of  slot  a  are  norvero,  then  the  slot  contains  the  result  returned  by 
one  of  the  computations  At  the  completion  of  the  other  computation  of  the  pair 
associated  with  that  continuation,  the  sum  of  its  value  and  the  contents  of  slot  a 
will  be  returned  to  the  higher-level  continuation  referenced  from  slot  c  thus  the 
algorithm  followed  by  each  continuation  object  is  the  same,  even  though  the  state 
Information  in  slots  a  and  c  varies  from  one  continuation  to  another  VIM  allows  us 
to  place  in  slot  t  of  a  target  object  a  reference  to  another  object  containing  the 
algorithm,  thus  avoiding  the  need  to  explicitly  copy  the  algorithm  into  each  of  our 
continuation  objects 
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figure  ?  10(e)  Continuation  of  Fibonacci  example 


All  the  remaining  active  calls  to  the  Fibonacci  actor  have  n  <  ?,  and  therefore 
cause  a  different  flow  of  control  through  the  actor's  type  object,  through  the 
ncwcv  request  in  slot  5  The  event  created  by  this  request  simply  returns  1  to 
the  continuation  specified  in  the  event,  furnishing  the  basis  for  our  recursion  F  ig- 
ure  ?  10(e)  shows  the  result  of  executing,  in  any  order,  the  three  remaining  active 
calls  to  the  Fibonacci  actor  Fach  generates  a  specific  event  returning  1  to  its 
continuation  Note  that  every  event  from  which  it  would  be  possible  to  reach  the 
main  body  of  the  Fibonacci  actor  is  now  inactive  Not  only  could  all  these  inactive 
events  be  reclaimed  by  a  garbage  collector,  but  since  part  of  the  Fibonacci  actor 
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is  now  inaccessible  (unless  it  Is  referenced  from  other  events  not  shown),  the  two 


objects  associated  with  this  part  of  it  cou'd  also  be  garbage-collected  I  he  code 
object  on  the  right-hand  side,  however,  Is  referenced  by  continuation  objects  that 
are  still  accessible,  and  hence  must  be  retained  for  now 
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Figure  2.10(f)  Continuation  of  Fibonacci  example 


In  Figure  2  10(f),  the  two  events  returning  to  the  rightmost  continuation  have 
concurrently  started  execution,  and  each  has  rea  hed,  but  not  yet  begun  to  exe¬ 
cute,  the  locktext  request  in  slot  1  of  thou  type  object  Both  executions  having 
obtained  (through  the  standard  start-up  sequence)  gtext  access  to  their  target 
object,  and  both  having  determined  that  slot  a  in  that  object  contains  jero,  each 
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will  now  attempt  to  lock  the  object  and  write  its  result  (found  in  slot  f  of  the 
event)  into  slot  a  of  the  target  object 
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f  igure  7  10(g)  Continuation  of  Fibonacci  example 


Figure  7  10(g)  shows  the  result  of  executing  the  two  locktext  operations 
Since  the  object  being  locked,  namely  the  target  object,  is  an  object  to  which 
gtext  access  had  already  been  obtained,  the  access  requests  shown  in  Figure 
7  10(g)  are  gtext-to-locktext  conversion  requests,  rather  than  simple  locktext 
requests  If  the  requests  were  simple  locktext  requests,  one  of  them  could  be 
granted,  execution  of  that  event  could  complete,  then  the  other  request  could  be 
granted,  and  execution  of  that  event  could  complete  Fach  event  would  write  a  1 
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into  slot  a  of  the  target  object  Ihis  sequence  ot  operations  would  be  wrong, 
since  the  second  event  to  complete  would  see  the  contents  ot  slot  a  change  dur¬ 
ing  its  execution,  an  occurrence  inconsistent  with  any  sequential  execution  of 
events  and  hence  in  violation  of  one  of  the  advertised  properties  of  VIM 

The  root  cause  of  this  problem  Is  that  in  seeking  to  upgrade  their  gtext 
access  to  locktext  access,  the  events  would  have  replaced  a  gtext  access 
privilege  with  a  locktext  access  reguest  Thus  an  access  privilege  would  have 
been  relinquished  before  completion  of  an  event  execution,  which  should  never 
happen  What  is  needed  is  something  that  combines  the  properties  of  the  gtext 
access  privilege  already  obtained,  and  the  locktext  access  request  now  being 
made  In  particular,  this  hybrid  should  prevent  locktext  access  from  being  awarded 
to  any  other  requestor,  for  this  would  be  inconsistent  with  the  gtext  access 
already  enjoyed  by  the  original  requestor  The  gtext-  to-locktext  conversion 
request  is,  in  fact,  just  this  combination,  and  Rule  H 7  correctly  indicates  that,  in 
Figure  ?  10(q).  neither  gtext- to-locktext  conversion  request  can  be  granted 
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The  only  altern.v.ive,  then,  is  to  abort  an  event  In  Figure  2.10(h),  the  upper 
event  has  been  aborted,  whereupon  the  lower  event  was  able  to  obtain  locktext 
access  to  its  *arpet  object  and  execute  the  side  effect  specified  in  slot  2  of  its 
type  object  Note  that  slot  a  of  the  target  object  now  contains  a  1  and  that  as  a 
result  the  side  effect  flag  of  thr?  event  has  been  set  to  true  This  means  that  the 
lower  event  can  no  longer  be  aborted 
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f  igure  ?  10(i)  Continuation  of  Fibonacci  example 


In  Figure  ?  10(i).  the  upper  event  has  again  begun  to  execute,  but  has  been 
stymied  early  in  its  start-up  sequence  by  its  inability  to  obtain  gtext  access  to  its 
target  object  This  access  cannot  be  obtained  because,  under  Rule  R6,  It  would 
conflict  with  the  iocktext  access  already  hel  1  fy  the  lower  event  This  lower 
event  can  no  longer  be  aborted,  since  it  has  performed  a  side  effect,  so  the  upper 
event  wdl  just  have  to  wait  until  execution  of  the  lower  event  completes  success¬ 
fully 
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In  the  transition  to  f  igure  2  10(j),  execution  ot  the  lower  event  has  completed, 
allowing  execution  of  the  upper  event  to  proceed  This  execution,  finding  the  con¬ 
tents  of  slot  a  In  the  target  object  to  be  nonzero,  proceeds  to  the  newev  opera 
tion  In  slot  4  of  the  type  object  The  event  created  by  this  operation  contains  in 
its  slot  f  the  sum  of  r:a  and  t:f.  which  in  this  case  is  the  value  of  f  that  was 
d  >sired 
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figure  2  10(k)  f  mal  configuration  for  f  ibonccci  example 


Figure  2  10(k|  shows  a  final  configuration  resulting  from  the  execution  of  the 
two  active  events  in  figure  2  10(j)  I  tie  value  of  f  from  the  upper  of  these  two 
events  has  been  copied  into  slot  a  of  its  target  object,  by  the  same  mechanism 
illustra *ed  in  figures  2  10(f)  through  (h)  The  one  remaining  active  event  in  figure 
2.10(k)  contains  the  sum  of  the  re-  ults  contained  in  the  two  active  events  of  Fig¬ 
ure  2  10(j).  and  in  fact  gives  the  correct  a' swer  f  }  =  3  ‘fo’e  that  at  this  point 
the  only  items  that  remain  accessil  le  are  this  one  event  and  the  caller's  original 
continuation  All  other  items  in  the  snapshot  can  be  garbage-collected,  unless 
referenced  from  other  active  events  not  shown 
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2.5.4:  Programming  In  VIM 

In  spite  of  the  f  Ibonacci  example  just  given,  the  reader  may  t>e  forgiven  for 
wondering  whether  (and  how)  it  is  possible  to  translate  arbitrary  programs  written 
Iri  other,  more  ordinary  programming  languages  Into  VIM  programs  An  existence 
proof  that  such  translations  are  possible,  at  least  from  a  programming  language 
similar  to  C[21]  is  the  compiler  for  programs  written  in  the  MuSpeak  language, 
which  generates  code  that  actually  runs  on  the  MuNet  The  reader  is  referred  to 
Strovink[34  35]  for  a  description  of  MuSpeak  and  for  a  much  deeper  discussion  of 
translation  issues  than  it  is  possible  to  give  here  In  this  section,  however,  we 
give  an  overview  of  the  complications  that  result  from  the  rules  of  the  VIM  inter¬ 
preter  and  ot  how  these  complications  may  ho  dealt  with 

One  complication  is  the  need  to  request  access  to  data  objects  before  actu¬ 
ally  operating  on  them  This  calls  for  a  modicum  of  planning,  so  that  all  necessary 
access  privileges  will  have  been  obtained  before  an  operation  is  attempted  This 
planning  is  complicated  further  by  the  fact  that  access  requests  (end  other 
resource  requests)  by  an  event  execution  are  illegal  after  a  side  effect  has  been 
performed 

Those  properties  of  VIM  require  every  computation  to  he  organized  into  quanta, 
where  each  quantum  t  rst  obtains  relevant  a*. cess  privileges  and  then  performs  any 
necessary  side  offer  ts  It  would  mdeod  be  feasible  in  some  cases  to  organize  an 
entire  program,  or  at  least  a  whole  subroutine,  into  one  such  quantum,  accumulating 
first  all  access  privileges  that  will  t  e  needed,  and  then  in  effect  executing  the  pro¬ 
gram  Optimization  questions  arise  here  on  event  accumulating  a  large  number  of 
access  privileges  and  taking  a  long  time  to  complete  is  more  likely  to  be 
aborted  but.  more  seriously,  the  single-guantum  approach  is  in  general  net 
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applicable  because  of  other  constraints  imposed  by  VIM 

One  such  constraint  is  the  time  bound  on  execution  of  an  event  once  it  has 
performed  a  side  effect  If  a  program  contains  loops  of  indefinite  duration,  such 
loops  must  be  broken  up  into  multiple  event  executions  to  ensure  that  no  event 
execution  continues  forever 

Another  factor  favoring  the  d»  composition  of  programs  into  smaller  quanta  is 
the  impossibility,  in  VIM,  of  "suspending"  execution  of  an  event  in  the  sense  that 
execution  of  a  routine  in  a  procedure-oriented  programming  language  may  be 
thought  of  as  “suspended"  during  a  i  all  to  a  subroutine  It  a  subroutine  call  is  to 
bo  encoded  as  an  event,  which  is  a  reasonable  thing  to  do,  then  the  body  of  the 
caller  must  be  split  into  at  least  two  quanta  one  which  contains  preparation  for 
the  call  and  culminates  in  creation  of  the  subroutine-call  event  itself,  and  one  which 
contains  the  processing  that  should  follow  a  return  from  the  subroutine 

Since  in  VIM  there  is  no  implicit  notion  of  a  "return"  to  the  previous  activity 
when  execution  of  a  subroutinr  has  completed,  an  explicit  i  ontrnuaf ion  object  must 
be  passed  along  with  every  subroutine  i  all  (as  was  done  with  the  Fibonacci  subrou¬ 
tine  in  our  example)  Invocation  of  the  continuation  as  a  target  object  should 
cause  the  resumption  of  the  routine  being  returned  to;  that  is,  t  should  c  nuse  the 
execution  of  the  mstrin  tions  following  the  procedure  i  all  to  which  the  continuation 
was  supplied  Thus  the  continual  in  object  will  be  the  second  of  the  two  objects 
discussed  at  the  end  ot  the  preceding  paragraph  II  ttie  procedure  being  called 
returns  one  or  more  v.Puos,  tlies"  r  a>  be  included  a-,  para  meters  in  tlio  event  that 
names  the  continuation  as  its  target  object  These  results  will  then  he  available 
during  execution  of  the  :or  tinuation  object 

The  considerations  discussed  above  concern  differences  In  control  structure 
between  VIM  and  other  languages  The  object-orientation  of  VIM  also  makt  s  fo» 
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differ  1'iiu's  m  d.if.i  strut  turn 


for  example,  VIM  has  nothing  quite  like  a  stack. 


which  is  a  concept  relied  upon  In  the  implementation  of  many  programming 
languages  fortunately,  VIM's  version  of  "heap"  storage  is  general  enough  to  model 
stacks  or  virtually  any  other  kind  of  data  structure  It  a  language's  environment 
structure  is  oriented  towaid  frames  on  a  stack,  each  such  frame  can  be 
represented  ns  a  VIM  object  Static,  dynamic,  and  any  other  desired  links  can  be 
handled  easily  by  using  object  references  The  VIM  targetobject/type-object 
mechanism  can  be  used  to  make  closure  objects  that  bind  together  a  piece  of  exe¬ 
cutable  code  and  an  environment  for  its  execution  Such  closure  objects  can  be 
used  for  passing  and  rotumn  g  procedure  parameters,  thus  VIM  is  fully  capable  of 
handling  oven  this  relatively  exotic  operation 

VIM'--  lack  of  explicit  support  for  stacks,  and  consequent  reliance  on  linked 
lists  of  objec  ts.  undoubtedly  has  some  cost  in  run-time  overhead  However,  in  sys¬ 
tems  wdh  many  co-operating  tasks  executing  concurrently,  such  as  the  multiproces¬ 
sor  systems  for  which  VIM  is  designed,  a  stark  is  quite  an  inadequate  environment 
strui  turn  anyhow  The  environment  structure  associated  with  such  situations  will 
generally  he  that  of  a  tree,  with  an  active  computation  associated  with  each  leaf 
node,  ami  various  parts  of  the  tree  "trunk"  shared  by  tasks  whose  common  ances 
tor  used  that  environment  linked  lists  of  objects  are  a  very  reasonable  way  of 
implementing  this  kind  of  environment  structure,  at.d  probabiy  cost  little  more  than 
"spaghetti  stacks"[P]  or  other  ways  rf  achieving  the  same  semantics 

Back  in  the  realm  of  control  structures,  the  notion  of  a  continuation  object, 
introduced  above,  ts  a  key  concept  for  handling  .ill  softs  of  problems  with  translat¬ 
ing  programs  into  the  VIM  style  f  ssentially,  if  one  can  envision  a  program  as  a 
flowchart,  such  that  the  operation  in  each  box  takes  a  bounded  amount  of  time  and 
never  needs  to  obtain  access  to  additional  data  after  performing  a  side  effect  (r.e., 
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such  that  the  operation  in  each  box  conforms  to  the  VIM  rules  for  a  legal  event 
execution),  then  the  translation  into  VIM  Is  simple  A  different  continuation  object 
can  be  generated  to  implement  the  operation  in  each  box,  and  then  only  two 
details  remain  to  be  settled  One  is  ttio  stringing  together  of  the  continuations, 
using  object  references,  so  that  every  continuation  fias  a  reference  to  each  suc¬ 
cessor  to  which  it  might  need  to  transfer  control,  the  other  is  the  provision  of  a 
suitable  execution  environment  tor  each  continuation,  which  may  be  done  either 
using  the  VIM  closure  mechanism  or  t>y  explicitly  passing  the  environment  to  the 
continuation  whenever  it  is  invoked 

Using  the  flowchart  analogy,  it  is  easy  to  see  how  programs  contaimrr;  itera 
tir.n.  conditionals,  sequencing  of  statements,  and  other  control  structures  can  be 
cot.  .true  ted  out  of  the  simple  box  -s  that  VIM  allows  There  is  ‘  till  a  groat  deal  of 
room  for  choice  ,n  determining  exactly  what  functionality  to  pack  into  each  box.  as 
well  as  just  how  to  organize  colie  *10115  of  objects  to  represent  (particular  data,  and 
how  to  implement  the  commutm  allot >  t imony  continuations  m  a  (program  These  gues 
tions,  whose  answers  <  an  have  sigmh<  ant  pet f ormance  lamifu  ations,  are  con¬ 
sidered  at  length  by  Strovink[34.3.‘  ]  Whatever  optimization  decisions  may  be 
made  in  the  course  of  a  translation,  though,  it  r  I  'ar  that  VIM  is  sufficiently 
powerful  to  serve  as  the  basis  fo.  implemen’ation  of  real  programming  languages, 
not  only  in  the  Turmg-completeness  sense,  but  in  r  ve.y  practical  sense  as  well 
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2.6:  Changes  of  Object  Format 

Given  the  fact  that  VIM  objectr.  may  persist  for  long  periods  of  time,  and  come 
with  all  kinds  of  different  lengths  and  sets  of  slot  identifiers,  it  is  convenient, 
though  not  logically  necessary,  to  be  able  to  change  the  format  of  an  object  Such 
a  facility  in  fact  exists  on  the  MuNet,  and  is  in  fact  useful  there  Changing  the 
Identifier  of  one  or  more  slots  in  a  blackboard  model  object  introduces  no  new 
complications  it  can  bo  treated  simply  as  a  side  effort  to  that  object  and  accord¬ 
ingly  handled  in  the  routine  Nl  X  T  -  INST  HlIC  T  ION  Changing  the  length  of  an  object, 
howove. ,  leads  to  some  subtle  ramifications  that  are  worthy  of  discussion 

Shortening  of  nr.  object  text  (,.e.  moving  Its  end  marker  to  the  left)  can  once 
again  be  viewed  as  a  •~lmj>le  side  effect  to  the  object  and  handled  by  existing 
mechanism  in  Nl  XT-INS 1 HUC  I  ION  If  an  object  is  to  be  lengthened,  though,  not  on'y 
a  side  effect  but  also  a  resource  request  (tor  more  sjrace  for  the  object  text)  are 
potentially  involved  Unfortunately,  a  resource  request  cannot  occur  after  a  side 
effect  has  been  performed  (for  tfien  it  would  be  impossible  to  abort  the  requesting 
event)  Thus  an  object  expansion  would  always  have  to  be  the  first  side  effect  in 
the  execution  of  an  event  (the  resource  request  would  have  to  occur  before  the 
actual  side  e'rpr.t)  This  rules  ou*  expanding  two  objects  in  one  event  execution, 
and  is  generally  an  annoying  constraint  on  the  construction  of  VIM  programs 

A  better  solution  is  to  split  the  object  expansion  operation  into  Its  two  com¬ 
ponents  flip  resource  request  and  the  side  effect  An  event  exec  ution  could  then 
request  additional  space  for  all  those  objec  ts  whose  expansion  was  contemplated, 
and  only  then  perform  the  actual  side  effects  Adoption  of  tins  strategy  requires  a 
modification  to  the  blackboard  interpreter  so  that  it  is  capable  of  representing  an 
object  for  which  additional  space  has  been  granted,  but  which  has  not  necessarily 
yet  open  expanded  to  use  that  space  Consequently,  the  concept  of  the 
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authorized  and  of  an  object  is  introduced,  as  illustrated  in  figure  2.11  1  he  total 


number  of  slots  in  an  object,  from  its  beginning  to  its  authorized  end,  is  referred  to 
as  the  object’s  author i /erf  length 

end  markur 


1 


•'I  [*>] 

3  I  b 

•r  j-i  | 

r  1  n|  __ 

_ 

— 

_ 

authorized  end 

figure  2  I  1  Object  wth  authorized  end  after  end  marker 

Only  the  slots  to  thu  left  ot  an  object's  end  marker  are  considered  semanti¬ 
cally  part  of  the  object,  slots  between  the  end  marker  and  the  authorized  end  are 
Inaccessible  and  have  unspecified  contents  It  is  assumed  that  when  the  side 
effect  of  actually  moving  the  end  marker  rightward  is  performed,  identifiers  and  ini¬ 
tial  contents  for  the  slots  moved  Jinst  will  be  specified  Conversely,  when  an  end 
marker  is  moved  leftward,  the  '  lots  moved  past  become  inaccessible  and  their  con¬ 
tents  become  unspecified  (tint  the  authorized  end  of  tt  e  object  does  not  move,  an  1 
remains  at  least  as  far  right  as  the  orig'nal  position  of  the  end  marker) 

There  is  no  point  in  asking  for  additional  slots  to  be  authorized  for  on  object 
unless  moving  the  end  marker  is  actually  intended  Ibis,  in  turn.  Is  a  side  eflect, 
for  which  locktext  type  access  is  .<  giired  Thus  it  makes  sense  for  the  primitive 
(which  we  call  objxpnd)  authorizing  additional  slots  for  an  object  to  also  perform  c 
locktext  request  for  the  object  This  decision  serves  another  purpose  as  well  If 
any  object  that  has  been  objxpndod  bu*  possibly  has  not  yet  had  its  end  marker 
moved  is  pointed  to  by  a  locktext  type  access  privilege,  then  the  absence  of  such 
a  privilege  pointing  to  an  object  may  be  taken  as  an  indication  that  the  extra  skits 
between  the  en  1  marker  and  the  authorized  end  are  no  l  inger  needed  and  may,  if 
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desired,  be  reclaimed  by  the  storage  system  this  consideration  leads  us  to  add 


one  more  basic  rule  to  the  blackboard  Interpreter 


Rule  R8:  If  the  object  o  has  no  locktext-  or  newobj-type  access  privileges  (solid 
arrows  with  sorifed  double  or  triple  arrowheads)  impinging  on  it,  move  the 
authorized  end  of  o  rightward  to  the  end  marker  of  o  This  rule  states  that, 
except  when  an  object  is  newly  created  or  Is  currently  susceptible  to  being 
written  into  (by  virtie  of  an  outstanding  tocktext  privilege),  its  authorized 
length  need  not  hi  preserved 


Additionally,  we  must  add  a  clause  to  Nl  XT-INSTRUC  TION  to  handle  objxpnd  calls 

objxpnd(ob;e--f,s/.-e)  li  SUM -l  f  f  l  C  T-PF  Rf  OBMl  (Kevenf )  then  FHROH.  else  if  HAS- 
LOCMFXT(#'vw/>f, object)  is  false,  then  LOCKTF  XT  (event  .object),*  else  advance  r 
in  the  activation  record  for  event  to  the  next  instruction,  and  increase  the 
authorized  length  of  object,  if  necessary,  to  be  at  least  as  large  as  si/e 

Beyond  these  changes,  Nl  »  T-INSTBIK'  TION  need  only  be  updated  to  stipulate  that 
moving  an  object's  end  marker  is  considered  a  write  into  the  text  of  that  object, 
and  to  Incorporate  the  obse> nations  made  above  concerning  the  disposition  of  slots 
added  to  or  removed  from  the  accessible  portion  of  an  object  text  by  moving  its 
end  marker 


*This  is  a  "hack"  to  make  sure  that  object  has  been  locktexted  before  Its 
authorized  length  is  changed  The  Intent  is  that  If  HAS-LOCKTEXT  returns  fa'Se, 
locktext  permission  will  be  established  and  then  the  objxpnd  request  re-executed 
This  is  why  the  program  counter  w  is  no!  incremented  until  after  locktext  permis¬ 
sion  has  been  established  The  Inelegance  ot  this  definition  does  not  correspond  to 
any  obscurity  in  the  concept,  nor  does  it  seem  to  have  foreboded  any  difficulties  in 
implementation 
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2.7:  Summary 


VIM  recognizes  two  basic  kinds  of  entities:  objects  and  events  Objects  may 
contain  arbitrary  binary  data,  and  may  also  freely  reference  other  objects  An 
event  is  a  request  to  execute  machine  code  found  in  the  type  object  of  the 
event's  target  object  An  event  may  call  upon  VIM  to  create  new  events  or 
objects  or  gain  access  to  object  texts  In  either  read-only  or  read/write  mode 
However,  any  event  execution  (1)  must  be  able  to  complete  in  finite  time  after 
having  performed  its  first  side  effect,  and  (?)  may  tie  aborted  at  «ny  time  prior  to 
performing  its  first  side  effect  No  event  execution  may  request  any  resource 
after  having  performed  a  side  effect,  as  defined  by  the  blackboard  interpreter 
thus  execution  of  an  event  consist-  of  two  phases  an  initial  phase,  when 
resources  are  accumulated,  texts  may  be  read  but  not  written,  and  now  objects 
and  events  may  be  created,  followed  by  a  final  phase,  which  begins  with  the  first 
side  effect  to  the  text  of  a  previously  existing  object  During  the  final  phase,  no 
additional  resources  may  be  requested  *  When  an  event  execution  terminates  suc¬ 
cessfully.  all  events  and  objects  created  by  it  are  formally  Installed,  and  all  access 
privileges  accumulated  by  it  are  terminated 

The  attractiveness  of  VIM  as  a  virtual  machine  for  parallel  processing  derives 
from  several  desirable  features  None  of  these  capabilities  is  all-important,  but 
each  contributes  to  the  appropriateness  ot  the  virtual  machine  for  distributed  com¬ 
puting,  and  VIM  provides  a  nice  package  for  them  all  First,  the  gtext-locktext 
style  of  object  access  not  only  lends  itself  naturally  to  the  solution  of  synchroniza¬ 
tion  problems  such  as  that  handled  hy  ‘lie  continuation  object  in  the  Fibonacci 
example  of  Section  ?  5  3,  but  gives  an  early  signal  of  which  resources  will  be 

*The  final  phase  need  not  occur,  it  is  entirely  reasonable  for  an  event  execu¬ 
tion  to  perform  no  side  effects 
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required  for  a  particular  event  execution,  allowing  the  system  to  assemble  them  in 
a  suitable  location.  It  becomes  unnecessary,  for  example,  to  give  every  processor 
in  the  network  direct  access  to  every  memory,  because  such  memory  accesses  as 
may  bo  necessary  during  execution  of  an  event  can  all  be  arranged  to  happen 
locally  This,  in  turn,  allows  reasonable  performance  to  be  exhibited  even  by  a  sys¬ 
tem  using  simple  and  low-cost  technology  in  its  communication  hardware 

A  consequence  of  this  VIM  object  access  style  is  that  a  program  must  contain 
checkpoints  to  which  execution  can  be  rolled  back  if  conflicts,  or  deadlocks, 
develop  Such  checkpoints  are  also  useful  for  indicating  points  at  which  moving  a 
task  from  one  processor  to  another,  or  performing  a  housekeeping  chore  such  as 
garbage  collection,  will  be  especially  easy  Implementations  of  VIM  are  simplified 
by  the  legitimacy  of  rolling  tasks  back  to  their  most  recent  checkpoints  when  such 
needs  develop 

The  VIM  concept  of  "event"  (in  the  concrete  sense  of  “event"  as  a  request  to 
perform  a  computation)  is  a  natural  manifestation  of  such  checkpoints  An  event  is 
also  a  compact  a-id  concise  package,  highly  suitable  for  shipme  it  between  proces¬ 
sors  containing  all  the  information  necessary  for  furthering  the  computation  it 
belongs  to  Finally,  the  use  of  events  In  the  style  allowed  by  VIM  not  only  makes  it 
very  easy  to  express  large  degrees  of  parallelism,  but  also  permits  the  construc¬ 
tion  of  highly  flexible  and  sophisticated  control  structures,  as  described  by 
Hewitt[  18) 

The  VIM  object  structure  similarly  ullows  the  construction  of  flexible  and 
sophisticated  data  structure.;,  of  the  sort  made  popular  by  users  of  LISP[26],  fur¬ 
thermore,  it  permits  data  to  be  represented  as  an  interrelationship  of  small  modules, 
enabling  operations  to  proceed  even  in  the  presence  of  only  parts  of  the  data 
Tins  capability  is  important  where  a  data  structure  may  be  flung  across  several 
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processors  In  a  system 


the  synergy  of  these  var.ows  features  makes  VIM  a  powerful,  simple,  and 
potentially  efficient  virtual  machine  particularly  suitable  for  implementing  modern 
object-oriented  languages  such  as  USP[?bJ  or  CLU[?4]  Services,  such  as  gar¬ 
bage  collection,  required  by  these  languages  are  implicitly  part  of  every  VIM  imple¬ 
mentation,  simplifying  the  job  of  generating  run-time  systems  VIM  objects  are 
flexible  enough  to  be  used  directly  to  represent  objects  in  those  languages  without 
undue  wastage  or  ineffn  .ency  finally,  VIM  events  are  quite  capable  of  represent¬ 
ing  the  computations  that  must  be  performed,  including  any  desired  use  of  parallel¬ 
ism 
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The  foregoing  discussion  of  VIM  describes  an  environment  capable  of  support¬ 
ing  message-passing  computation  in  a  fairly  elegant  style,  however,  its  usefulness 
in  the  broader  context  of  a  computer  utility  can  be  enhanced  by  the  addition  of  a 
few  extra  features  I  he  purpose  of  this  chapter  Is  to  outline  extensions  that 
create  a  more  hospitable  environment  for  operating  systems  running  under  VIM,’ 
readers  more  Interested  in  other  topics  may  skip  this  chapter  with  no  loss  of  con- 
t'liuity  Elaborations  or  some  of  these  extensions  are  also  given  in  Appendix  A  A 
reference  tree  network  could  be  useful  in  practice  even  without  these  extensions, 
the  purpose  of  the  following  discussion  is  to  demonstrate  the  power  and  flexibility 
of  the  VIM  approach,  in  that  desirable  system  features  can  bo  incorporated  without 
doing  violence  to  the  essential  concepts  of  VIM 

In  the  context  of  VIM,  an  "operating  system"  might  servo  several  functions  It 
might  provide  for  long-term  storage  of  user  data,  allow  for  user  control  and  tracing 
of  program  execution,  or  mediate  allocation  and  usage  of  various  kinds  of 
resources  For  maximum  flexibility,  it  is  desirable  to  avoid  building  such  an  oporat- 
ing  system  into  VIM  A  preferable  aporoach  is  to  supply  only  the  basic  "hooks" 
necessary  to  enable  a  user  to  implement  his  own  operating  system  services.  The 
architecture  of  an  operating  system  built  around  this  philosophy  is  described  by 
Gula[  13,14],  our  purpose  here  is  to  concentrate  on  the  "hooks"  in  VIM  necessary 
for  support  To  begin  with,  we  concentrate  particularly  on  a  feature  which  facili¬ 
tates  user  augmentation  of  the  basic  VIM  environment,  and  also  provides  additional 
control  over  execution  of  user  programs. 

"For  a  discussion  of  such  systems,  see  [13,14] 
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3.1 :  Event  Tracing 


In  the  primitive  VIM  environment,  once  a  program  has  started  running  (by  crea¬ 
tion  of  some  initial  event)  the  only  control  that  can  be  exercised  over  that 
program's  execution  is  that  built  into  the  program  Itself  If  the  program  goes  Into 
an  infinite  loop,  there  is  no  way  to  force  its  execution  to  be  terminated  It  is  pos¬ 
sible.  within  the  primitive  VIM  contoxt,  to  adopt  a  convention  for  creating  events 
wherein  a  specific  "trace  actor”  gains  control  before  each  new  event  execution 
The  trace  actor  will  then  determine  whether  to  proceed  normally  with  the  event 
execution  or  whether,  for  ex  imple,  to  terminate  that  chain  of  events  or  invoke 
some  trace  routine  before  continuing  Unfortunately,  if  a  user  fails,  either  acciden¬ 
tally  or  on  purpose,  to  obey  tins  convention,  execution  of  his  program  will  not  be 
controllable  by  this  means  Another  objection  to  the  proposal  is  that  it  introduces 
substantial  extra  overhead  to  make  a  decision  which  in  the  vast  majority  of  cases 
will  be  simply  "go  ahead  " 

A  better  solution  is  to  asso  late  with  each  event  a  process  object  which  as 
part  of  its  text  would  have  a  reference  to  such  a  trace  actor  In  the  normal  case, 
this  could  be  a  reference  to  some  distinguished  object  meaning,  simply,  "proceed  " 
In  the  context  of  the  blackboard  interpreter,  every  event  could  be  considered  to 
have  a  slot  labeled  will  the  reserved  identifier  containing  a  reference  to  the 
process  object  (see  Figure  3  1)  Before  executing  an  event.  VIM  would  examine 
the  reference  to  that  event's  trace  actor  (found  in  slot  #  of  the  event's  process 
object)  If  that  r  *ference  were  found  to  s<  y  '  proceed,"  normal  event  execution 
would  ensue  Otherwise,  the  trace  actor  would  be  invoked,  supplanting  the  normal 
invocation  of  the  event's  target  object  The  trace  actor  could  then  determine  the 
proper  course  of  action,  and  if  execution  of  the  event  were  called  for,  eventually 
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transfer  control  to  the  event's  target  object 


Procoss  object: 


if  : 

IT  1  " 

Lvent: 

| — ' 

Parameters 
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I V 
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Process  context 


I  race  actor 


T  racing  code 


Figure  3  1  Procoss  object  and  trace  actor 


By  default,  the  process  object  would  be  passed  down  from  an  ovont  to  its  suc¬ 
cessors  (new  events  created  by  it),  thus  assuring  continuity  of  the  tracing  mechan¬ 
ism  through  many  generations  of  events  Due  to  its  automatic  inheritance  by  newly 
created  events,  the  process  object  is  the  logical  repository  for  all  kinds  of  context 
Information  (such  as  user  identification,  privileges,  quotas,  etc  )  normally  associated 
with  a  process  in  a  more  traditional  operating  system  *  When  an  attribute  is  stored 
in  an  event’s  process  object,  it  will  automatical'y  apply,  unless  subsequently 
changed,  to  all  events  descended  from  the  original  event  that  is,  to  the  entire 
computation  initiated  by  the  first  event  This  notion  of  process  object  is  even  more 
ftoxible  than  the  usual  concept  of  process,  since  a  single  process  object  can  even 
tie  shared  by  several  concurrently  ex*?'  utir.g  events  but  presumably  only  if  these 


*ln  order  to  maintain  the  integrity  of  this  information,  it  would  probably  be 
desirable  for  VIM  to  put  some  restrictions  on  the  ability  to  change  an  event's  proc¬ 
ess  object 
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events  are  co-operating  toward  a  single  goal  identified  with  that  process  object 

A  method  for  structuring  and  using  process  objects  is  discussed  by  Gula[14j. 
for  our  purposes,  only  relatively  simple  details  surrounding  the  creation  of  a  seman¬ 
tic  task  (a  set  of  events  co-operating  toward  a  particular  goal)  are  of  interest. 
When  such  a  task  Is  initiated,  a  new  process  object  should  be  created,  and 
assigned  to  the  Initial  event  of  the  task  If  the  creator  of  the  task  retains  a  refer¬ 
ence  to  the  process  object,  the  creator  will  be  able  to  manipulate  the  task  by 
changing  values  in  *he  text  of  the  process  object  For  example,  if  it  is  desired  to 
cancel  the  task,  a  trac  e  actor  can  be  inserted  which  will  terminate  any  event  that 
invokes  it  The  ability  to  easily  perform  this  kind  of  manijiulation  is  the  reason  for 
introducing  the  extra  level  of  indirection  of  placing  the  reference  to  the  trace 
actor  within  the  text  of  a  process  object,  rather  than  having  it  be  directly  an  attri¬ 
bute  of  an  event  Changing  a  particular  process  object's  trace  actor  reference  will 
suffice  to  influence  all  events  using  that  process  object,  avoiding  any  need  to 
explicitly  know  the  identities  of  the  affected  events 

This  trace  actor  mechanism  is  perhaps  not  yet  powerful  enough  to  meet  all  rea¬ 
sonable  needs  Halting  a  particular  line  of  computation  is  made  simple  enough,  but 
ga'he-ing  execution  statistics  or  performing  an  event-by-event  execution  trace  is 
still  cumbersome  This  is  because  the  tra  :e  actor  will  have  to  explicitly  transfi  r 
control  to  the  event's  target  object  when  done  with  its  own  tracing  operations 
The  target  object  will  presumably  want  to  access  some  object  texts,  or  otherwise 
make  requests  that  might  result  in  the  event's  being  aborted  By  our  earlier  rules, 
such  requests  would  be  illegal  after  any  side  effect  had  been  performed  on  any 
object  text  Therefore,  tfie  trace  actor  must  ref'ain  from  performing  ,«ny  side 
effect,  such  as  incrementing  a  count  of  successful  event  executions,  if  it  is  subse¬ 
quently  going  to  transfer  control  to  the  event’s  target  object  The  only  way  the 
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trace  actor  can  leave  behind  a  record  of  a  successful  execution,  without  violating 
any  rules,  is  to  create  an  event  recording  the  occurrence.  That  event  will  only  be 
activated  if  the  subsequent  execution  of  the  target  actor  completes  successfully 
Assuming  the  event  is  indeed  activated,  it  can  then  trigger  an  arbitrarily  complex 
series  of  computations,  including  side  effects,  to  update  the  relevant  statistics 

Thus  statistic-gathering,  although  cumbersome,  is  not  impossible  I  he  operation 
which  remains  intractable  is  interactive  tracing,  event  by  event,  ot  a  program  exe¬ 
cution  Mere,  it  may  tie  desired  to  construct  a  trace  actor  which  pauses  before 
executing  an  event,  waiting  fur  user  input  before  continuing  this  much  ran  be 
achieved  using  tin-  trace  actor  mechanism  thus  fur  described  What  is  difficult  is 
installing  a  new  trace  actor  which  will  allow  the  current  event  to  continue,  but 
pause  before  t'xocutmg  any  descendant  eve-  /  created  by  the  current  one 

This  is  similar  to  the  problem  of  correctly  implementing  a  "trace  bit,"  specified 
to  cause  a  trace  interrupt  before  every  instruction  execution,  on  a  processor  of 
traditional  architecture  The  solution  is  also  similar  An  event  must  have  two 
states  not-yet-traced  and  already  traced  These  states  are  comparable  to  the 
active/inactive  s’atus  indicated  by  ni  event's  active  flag  When  a  not-yet-traced 
event  u  scheduled  for  execution,  its  trace  a  tor,  if  any,  is  invoked  instead  of  its 
target  object  Afte'  a  successful  trace  a<  tor  invocation,  the  event  is  changed  to 
the  already  tra-  ed  state  In  this  state,  the  target  object  is  always  invoked’  N>>w 
events  would  ordinarily  be  created  In  the  not-yet-traced  state,  for  proper  operation 
of  the  fracing  mechanism  However,  a  sjiecial  facility  for  creating  already-traced 


*lt  may  he  desirable  to  allow  the  interposition  of  a  short,  side-effei  t-free 
"affertrace  actor"  here  to  permit,  for  example,  immediate  killing  of  events  even  if 
they  have  reached  the  already- traced  state 
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events  would  tii*  useful  to  debugging  software 

In  summary,  several  advantages  can  be  gamed  by  associating  with  every 
event  a  proi  ess  ofc/e«  f  Ihe  process  object  is  useful  in  implicitly  passing  context 
information  down  through  a  whole  chain  of  successive  events  If  is  also  useful,  In 
conjunction  with  the  concept  of  the  fr<*»  e  Actor,  for  retaining  control  of  a  compute 
tion  that  has  been  started,  and  in  tracing  and  gathering  statistics  Its  usefulness 
in  this  regard  may  be  enhanced  by  distinguishing  between  already  traced  events 
and  not-yet-traced  events 

3.2:  Object  Tracing 

Another  service  which  an  operating  system  might  want  to  use  would  be  a  facil¬ 
ity  for  monitoring  patterns  of  access  to  object  texts  In  a  sense,  this  kind  of 
capability  Is  the  dual  of  the  event  tracing  function  described  above,  but  some  of 
the  difficulties  are  greater  In  any  case,  one  can  distinguish  an  analogous  set  of 
useful  operations  gathering  statistics  regarding  accesses  to  an  object,  intercept 
mg  such  accesses,  >r  suspending  them  to  allow  for  some  period  of  computation  or 
user  interaction  Thus,  hy  analogy,  one  might  imagine  associating  a  "trace  actor" 
with  each  object  Code  associate*  with  the  trace  actor  would  be  invoked  upon 
any  attempt  (via  gtext  or  locktext)  to  across  the  text  of  the  object  As  before, 
some  default  value  would  mean  "no  fracing,"  whereupon  access  to  the  text  of  the 
object  would  follow  the  normal  procedure  outlined  previously 

Gathering  statistics  regarding  accesses  to  an  object  would  be  possible,  sub¬ 
ject  to  restric  tions  similar  t)  those  Jixcussed  in  our  first  proposal  for  event  tiacmg 
no  side  ef  pets  could  he  direef iy  performed  by  an  object's  tiace  actor,  vnee  the 
Currently  executing  event  might  attempt  additional  resource  requests  after  return 
from  the  trail'  at. ter  the  trace  actor  could,  however,  create  a  new  event  to 
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record  the  tact  of  the  object's  having  been  accessed  It  is  true  that  an  attempt 
to  create  such  an  event  is  a  resource  request  which  might  result  in  abortion  of  the 
current  event,  but  this  will  never  cause  a  problem,  since  the  trace  actor  is  only 
Invoked  In  the  course  of  processing  a  request  which  might  in  itself  cause  the 
current  event  to  be  aborted 

Another  possible  use  of  a  traco  actor  for  an  object  would  be  in  validating 
requests  to  access  the  object  Under  VIM,  the  main  control  on  accoss  to  an  object 
is  possession  of  a  reference  to  the  object,  which  is  admittedly  quite  a  coarse 
restraint  A  trace  actor  could  examine  the  process  object  of  the  current  event  to 
determine  the  identity  ot  its  owner  On  tl  *•  basis  ot  this  information  it  could  decide 
whether  to  allow  access  to  the  object  in  the  requested  mode  If  access  wore  to 
■  •  domed,  the  trai  e  actor  could  terminate  •■xecution  of  the  current  event  hy  sig 
nailing  an  error  to  VIM  this  is  a  rattier  clumsy  prt  tertHin  mechanism,  though, 
further  dis,  uss  >n  of  protection  issues  will  appear  below 

A  third  use  for  trac  e  actors  would  be  in  conjunction  with  an  incremental  compi¬ 
lation  or  dynamo  linking  scheme  Unoompiled  or  "unlinked"  objects  could  be  given  a 
special  trace  actor  which  would,  upon  first  access  initiate  a  compilation  or  linking 
stej>  to  fill  in  the  corn*,  t  text  for  the*  ohjec  t  The  trace  actor  could  suspend  and 
record  all  requests  for  access  to  the  object  while  being  made  ready,  then  re¬ 
create  all  the  suspended  events 

Perhaps  th<*  most  important  application  of  object  trace  actors,  however.  Is  in 
restoring  a  certain  universality  of  message-pass. nq  systems  that  is  compromised  to 
some*  extent  (in  return  for  a  potential  gain  in  run-hme  efficiency)  in  the  design  of 
the  basic  VIM  machine  In  .1  pure  message  passing  system,  all  interaction  between 
entities  in  the  system  is  by  exchange  of  messages,  thus  any  module  in  the  sys¬ 
tem  can  he  fully  sj>e.  ified  by  detailing  its  response  to  various  different  message 
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protocols  I  ho  fact  that  the  only  way  to  communicate  with  a  modulo  is  by  passing 
It  a  message  makes  it  simpU?  to  transparently  splico  additional  processing  into  the 
message  path  ahead  ot  the  module  this  could  be  done  either  tor  the  purpose  of 
augmenting  or  altering  the  semantics  of  the  original  module  by  modifying  its 
response  to  certain  messages,  for  the  purpose  of  gathering  usage  statistics,  or  for 
the  purpose  of  creating  a  virtual  entity  within  ttie  system  An  example  of  this  last 
application  is  the  concept  of  a  fufure[3j.  in  which  a  dummy  actor  is  supplied  as  a 
place-holdor  to  collect  and  save  messages  received  while  the  real  actor  that  is  to 
receive  those  messages  is  in  preparation  Another  example  could  occur  in  the  con¬ 
text  of  virtual  memory  management  an  at  tor  could  be  “swapped  out"  and  a 
dummy  substituted  for  it  to  initiate  a  "swap  in"  operation  upon  receiving  a  mes¬ 
sage  At  any  rate,  mi  important  fact  abou*  the  exclusive  use  of  message-passing 
protocols  is  that  they  allow  an  arbitrary  computation  to  he  triggered  by  roi  eipt  ot  a 
message  This  is  the  universality  property  alluded  to  above 

In  VIM.  however,  there  aie  two  ways  of  interacting  with  an  object  One  is  oy 
sending  it  messages,  the  other  is  by  accessing  it  directly  (via  gtext  or  lodde-.t) 
The  message-passing  mode  of  interaction  has  the  nic  e  properties  discussed  above, 
but  the  accessing  mode  does  not  The  addition  of  object  tracing  to  VIM  restores 
these  universality  properties  Hy  specifying  an  appropriate  trace  actor  for  an 
object,  it  is  possible  to  triqger  an  arbitrary  computation  upon  any  access  to  the 
object  Thus  any  of  the  virtual  memory  or  other  meeh.-.msms  mentioned  in  the  prove 
ous  paragraph  can  he  implemented  This  general  capability,  in  far  *,  is  at  the  heart 
of  the  applicability  of  object  tracing  to  all  the  uses  discussed  earlier  in  this  sec 
Iron 
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3.3:  Protection 


In  the  basic  VIM  environment,  possession  of  a  reference  to  an  object  is  a 
grant  of  unlimited  access  to  that  object  and  all  objects  reachable  by  chains  of 
references  starting  with  that  object  I  here  are  many  situations  In  which  it  is  desir¬ 
able  to  give  only  partial  access  to  an  object,  for  example,  access  to  Invoke  the 
object  as  a  target  object,  but  not  to  read  or  write  its  text  One  might  wish  to  for¬ 
bid  writing  into  a  text  to  prevent  some  unauthorized  agent  from  damaging  the 
object,  one  might  wish  to  prohibit  reading  from  a  text  to  avoid  giving  out  refer¬ 
ences  contained  within  it 

The  trace  actor  mechanism  outlined  in  the  previous  section  may  be  used  to 
discriminate  among  the  access  rights  granted  to  different  holders  of  a  reference, 
provided  that  it  is  impossible  for  an  unprivileged  holder  of  a  particular  reference  to 
masquerade  as  a  privileged  one  It  can  also  bo  used  to  enforce  "execute-only" 
access  to  an  object  either  across  the  board  or  for  certain  users  Under  execute- 
only  access,  the  object's  text  can  be  act  eased  only  during  its  own  execution,  r.e  , 
if  d  is  the  target  object  of  the  current  event  A  user  distributing  a  reference  to 
an  execute  only  object  need  not  fear  compromising  the  secrecy  of  the  references 
contained  therein,  for  they  can  only  be  accessed  by  the  machine  code  associated 
with  the  execute-only  object  itselt,  which  can  apply  arbitrarily  complex  security  cri¬ 
teria  before  divulging  any  information 

Similarly,  the  trace  actor  mechanism  can  be  used  to  enforce  read-only  access. 
y,h.-rem  the  text  of  an  object  may  be  read  but  not  changed  In  fact,  across-the- 
board  imposition  of  read-only  or  execute-only  access  (or  both')  might  be  a  r-ommon 
enough  operation  to  v  arrant  adding  special  bits  (for  efficiency)  to  oh|ect  texts  for 
specifying  either  kind  of  access  restriction 

Combined  with  the  ability  to  generate  unique  references  ty  creating  new 
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objects,  and  use  these  references  later  as  keys,"  these  admittedly  act  hoc  protec¬ 
tion  mechanisms  might  actually  serve  as  the  basis  for  a  reasonably  comprehensive 
security  system  However,  research  on  capability  systems  tias  led  to  much  more 
elegant  solutions  to  various  protection  problems[28,4 1 J,  it  remains  to  be  seen 
exactly  how  best  to  use  these  ideas  in  the  VIM  environment 

3.4:  File  System  Support 

Another  area  of  operating  system  interaction  is  in  helping  the  user  manage  his 
data  Since  VIM  supports  a  universe  of  objects  with  unlimited  ability  to  reference 
each  other,  it  would  seem  natural  to  use  these  as  a  basis  for  building  any  desired 
filing  system  the  user  could  represent  his  data  as  objects,  and  references  to  any 
such  objects  could  he  entered  into  a  directory  system,  also  composed  of  objects 
Unfortunately,  tho  storage  of  reasonable  amounts  of  data  in  this  fashion  imposes  a 
requirement  on  the  network  to  maintain  a  largo  virtual  memory  space  of  objects, 
larger,  for  example,  than  could  hi  in  the  collective  primary  memories  of  the  proces¬ 
sors  in  the  network  Thus  low  level  network  software  might  be  required  to  page 
objocts  in  and  out  from  secondary  storage  attached  to  the  network  at  one  or  more 
nodes  Such  a  scheme  has  the  advantage  of  being  completely  transparent  to  all 
programs  running  on  the  network,  but  the  possible  disadvantages  of  being  harder  to 
control  explicitly  where  such  control  is  desired,  and  of  further  enla'gtng  and  compli¬ 
cating  the  lowest-level  network  software 

An  alternative  approach  is  to  leave  the  management  of  secondary  storage  to 
higher  levels  of  "operating  system"  software,  possibly  supported  by  somo  special- 
purpose  facilities  implemented  at  lower  levels  (  the  appropriate  facilities  are  pro- 

*The  term  “keys"  here  is  as  used  by  HendersonJ  1  7] 
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vided,  suet)  a  scheme  can  remain  reasonably  transparent  to  user-level  software 


(see  Gulal  14)). 

A  problem  that  must  be  faced  by  any  scheme  for  data  management  on  a  large 
scale  is  that  of  reliability  t  specially  it  viewed  as  a  multiprocessor  architecture, 
rather  than  a  distributed  computing  network,  it  may  be  reasonably  tolerable  for  a 
reference  tree  network  to  "crash"  occasionally  due  to  some  failure,  and  it  may  be 
convenient  to  bring  down  the  network  periodically,  for  reconfiguration,  maintenance, 
or  whatever  Such  a  “crash"  may  entail  the  interruption,  even  the  loss,  of  all  com¬ 
puting  then  occurring  on  the  network,  but  it  is  not  tolerable  for  it  also  to  entail  the 
loss  of  all  data  stored  in  the  system  Thus  whatever  large-scale  data  storage 
scheme  is  used  must  be  '  stable"  in  the  sense  of  being  able  to  survive  virtually  any 
conceivable  incident  on  the  system  substantially  intact  This  implies  that  updates 
to  data  in  "stable  storage"  must  be  recorded  in  a  non-volatile  fashion  within  a  rea¬ 
sonable  length  of  time  It  also  implies  that  when  the  system,  or  some  part  of  it,  is 
restarted,  the  non- volatile  record  must  be  available  and  understandable  These 
challenges  must  be  faced  by  any  software  that  implements  the  large-scale  data 
storage  mechanism,  whether  at  the  lowest  level  or  at  higher  levels 

One  possible  way  of  managing  stable  data  storage  without  building  it  into  VIM 
is  to  use  the  object  tracing  mechanism  One  "object  manager"  might  be  associated 
with  each  mass  storage  device  in  the  system  Inactive  objects  could  be  paged  out 
to  mass-storage  devices  and  ''aliases"  understandable  to  the  object  managers  sub¬ 
stituted  for  them,  every  such  alias  could  name  as  its  trace  actor  the  relevant 
object  manager  When  an  object's  text  is  again  needed,  a  "trap"  to  the  object 
manager  could  cans*-  it  to  be  retrieved  from  mass  storage 

Unfortunately,  accessing  efficiency  is  not  the  only  casualty  when  aliases  are 
introduced.  the  network  garbage-collection  mechanism  also  suffers  When  an 
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object  Is  converted  into  an  alias,  its  object  manager  must  record  all  references 
contained  in  the  object's  text,  for  use  if  the  object  over  needs  to  be  reconstituted 
out  of  the  alias  Since  these  objects  are  always  referenced  from  the  object 
manager,  they  cannot  be  deleted  unless  the  object  manager  Itself  becomes  inac¬ 
cessible  Ibis  In  turn  cannot  happen  unless  every  object  managed  by  the  manager 
becomes  Inaccessible  This  is  a  realistic  possibility  for  some  well-chosen  assign¬ 
ments  of  objects  to  managers,  but  is  by  no  means  likely  in  the  general  case 

3.5:  Summary 

This  chapter  described  possible  modifications  to  the  VIM  virtual  machine 
designed  to  facilitate  controlling  and  gathering  statistics  about  event  executions 
anil  object  accesses,  handling  protection  problems,  and  providing  other  operating 
system  services  A  reference  tree  network  could  t>e  quite  useful,  simply  ns  a  com¬ 
puting  engine,  even  without  any  of  these  facilities,  however,  the  incorporation  of 
features  such  as  those  described  would  contribute  to  making  a  network  a  more 
well-rounded  total  computer  system  I  or  the  purposes  of  this  thesis,  perhaps  the 
real  significance  of  the  material  discussed  in  this  chapter  lies  elsewhere  yet  it 
shows  some  of  the  flexibility  of  the  VIM  programming  style  in  adapting  to  the  addi¬ 
tional  needs  considered  in  this  chapter  This  floxib'lity  is  an  important  reason  for 
being  enthusiastic  about  VIM  as  a  good  Interface  between  programs,  operating  sys¬ 
tems,  and  multiprocessor  networks 
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Chapter  4:  Architecture  of  Reference  Tree  Networks 


4.1:  The  Physical  Machine:  Network  Topology 

As  much  as  possible,  it  is  the  intent  of  this  research  to  avoid  becoming  commit¬ 
ted  to  the  narrow  technological  characteristics  of  any  one  type  of  network  How¬ 
ever,  for  concreteness,  it  is  helpful  to  make  certain  assumptions  f  urthermore,  the 
algorithms  presented  depend  on  a  certain  loijn  j /  organization  of  processors  which 
may  be  more  easily  achievable  with  some  physical  organizations  than  with  others 

The  basic  logical  structure  assumed  by  our  Implementation  is  a  collection  of 
processors,  each  with  a  private  local  memory  (/.<•..  no  sharing  of  memory  between 
processors  Is  required),  each  connected  to  a  small  number  of  other  processors 
which  are  its  neighbors  Tin*  connections  are  bidirectional  and  symmetrical,  thus  if 
A  is  a  neighbor  of  B  then  B  is  a  neighbor  of  A  Such  an  organization  of  processors 
is  equivalent  to  an  undirected  graph  (which  may  or  may  not  contain  cycles)  in 
which  only  a  few  arcs  emanate  from  each  node  "few"  here  is  a  relative  term,  its 
use  is  due  to  the  fact  that  the  overhead  incurred  by  a  nrocessor  will  increase  if 
that  processor  accumulates  more  neighbors  thus  it  might  well  be  appropriate  for 
higher-capacity  machines  or  machines  with  more  of  a  commitment  to  serve  the  net¬ 
work  (rather  than,  say.  their  owners)  to  have  greater  numbers  of  neighbors  A 
variety  of  topologies  are  consistent  with  these  general  restrictions  (some  are 
shown  in  Figure  4  1) 

.'n  addition  to  the  specifications  given  above,  each  processor  is  required  to 
have  a  processor  ID  unique  to  the  entire  system  (This  ID  is  used  in  generating 
unique  names  and  unique  time  stamps  If  unique  names  and  time  stamps  can  be 
dispensed  with,  or  generated  in  some  other  fashion,  processor  ID's  are  not  neces¬ 
sary.)  It  is  not  necessary  that  all  processors  be  identical,  as  long  os  all  interpret 
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Figure  4  1  Some  possible  network  topologies 


substantially  the  same  machine  language 

The  physical  topology  of  the  system  need  not  follow  the  logical  topology 
described  above  The  logical  topology  in  effect  constrains  the  paths  over  which 
Information  may  travel,  any  physical  topology  which  permits  information  How  over 
these  channels  may  form  the  basis  for  an  acceptable  implementation  Tor  example, 
an  Flhernet  (on  which  every  processor  can  communicate  directly  with  every  other 
processor)  could  be  made  to  support  th}  logical  topolopy  descrl!  ed  above  either  by 
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declaring  every  processor  to  be  a  neighbor  ot  every  other  (which  might  however 
impose  a  large  amount  of  overhead  on  each),  or  by  choosing  for  each  processor  a 
set  of  logical  neighbors,  and  constraining  the  communication  patterns  on  the  net  so 
that  no  processor  ever  sends  a  message  to  another  processor  that  Is  not  its  neigh¬ 
bor  However,  tins  is  tar  from  the  Ideal  way  of  using  an  Ethernet. 

In  any  case,  the  scheme  presented  here  was  certainly  not  designed  for  Ether¬ 
nets,  but  rather  for  physical  networks  with  properties  closely  matched  to  those  of 
the  logical  network  Such  networks  have  several  advantages 

•  expansibility  The  network  can  be  expanded  to  a  very  large  si/e  at  little 
marginal  cost  The  space  allocated  for  unigue  processor  ID's  does  grow,  but 
only  logarithmically  Otherwise,  expansion  presumably  involves  simply  hook¬ 
ing  up  m*w  processors  to  the  edges  of  ttie  network,  and  has  only  a  very 
local  impact 

•  bandwidth  for  many  topologies,  there  are  potentially  a  largo  number  ot 
communication  paths  between  any  two  points  in  the  network,  no  central 
f  ther  or  other  medium  serves  as  a  bottleneck 

•  reliability  t  ven  a  catastrophic  hardware  failure  at  some  node  is  likely  to 
affect  only  a  limited  number  of  other  nodes  (its  neighbors)  In  systems  with 
a  central  medium,  there  are  components  whose  failure  will  stop  all  communi¬ 
cation  on  the  netvvork  Of  course,  reliability  is  also  strongly  influenced  by 
the  software  system's  ability  to  carry  on  in  the  face  of  failures,  admittedly, 
this  thesis  does  not  address  this  problem  very  thoroughly 
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•  flexibility  Many  diflerent  processor  and  link  technologies  can  In  principle 
coexist  in  the  system,  allowing  considerable  freedom  in  picking  the  lowest- 
cost  option  for  the  performance  desired  at  each  point 

There  are  disadvantages  to  this  kind  of  organization  also  Chief  among  these 
is  the  need  for  extra  processors  to  become  Involved  in  transactions  between  proc¬ 
essors  which  are  not  neighbors,  with  the  attendant  overhead  and  delay  It  is 
hoped  that  the  scheduling  strategies  proposed  in  Chapter  7  will  tend  to  minimize 
the  need  for  this  kind  of  transaction 

The  topologies  depicted  in  Figure  4  1  consist  of  units  which  are  private  proces¬ 
sor-memory  pairs,  connected  by  communication  lines  An  attractive  alternative 
might  be  composed  of  multiport  processor  and  memory  elements,  as  shown  in  Figure 
4  ?  Although  each  processor  might  have  its  own  private  storage  for  read-only  and 
temporary  data,  all  objects  would  be  stored  in  the  multiport  memories  Assuming  a 
nominal  amount  of  local  program  or  microcode  memory  at  each  processor,  it  should 
be  possible  to  connect  as  many  as  three  or  four  processors  to  each  memory 
without  serious  degradation  of  performance  due  to  access  conflicts  An  architec¬ 
ture  such  as  this  has  some  flexibility  advantages  a  processor  can  attach  itself  to 
any  adjacent  memory  that  contains  data  to  be  operated  on,  or  even  simultaneously 
operate  on  data  stored  in  several  diflerent  memories  Additionally,  each  processor 
can  serve  as  an  active  communication  link  between  any  of  its  adjacent  memories, 
moving  data  from  one  to  another  (or  even  performing  more  sophisticated  operations) 
at  high  speed  For  a  tightly  coupled  local  system,  this  multiport  architecture  is 
very  attractive 

The  reference  tree  network  implementations  we  will  discuss  are  described  in 
terms  of  the  kinds  of  topologies  shown  in  Figure  4  1  They  can  be  adapted  to  mul- 
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Figure  4?  Multiport  processor-memory  networks 


tiport  networks  such  as  those  in  Figure  4  2  by  considering  each  memory  to  be  o 
node,  and  drawing  links  between  every  pair  of  memories  that  shore  a  common  proc¬ 
essor.  Some  modifications  to  the  reference  tree  algorithms  would  undoubtedly  help 
make  the  best  use  of  the  capabilities  of  multiport  networks,  but  the  basic  concepts 
are  quite  applicable  to  either  kind  of  network 
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I  ho  static  anti  structural  aspects  of  our  implementation  have  now  been 
described  in  sufficient  detail  that  we  can  proceed  to  consider  its  dynamics,  that  is, 
the  motivation  and  mechanism  surrounding  the  scheduling  of  events,  sending  of 
messages,  and  so  forth  In  the  remainder  of  this  thesis,  we  shall  generally  be  con¬ 
cerned  witti  a  "steady-state"  situation  in  which,  as  a  result  of  some  unspecified 
history,  several  processors  in  the  system  have  been  assigned  tilings  to  do,  and 
need  to  communicate  with  other  processors  containing,  for  whatever  reason,  data 
upon  which  they  need  to  operate  In  this  section,  therefore,  we  shall  concentrate 
on  developing  some  intuition  for  such  a  "steady-state"  situation,  briefly  exploring 
mechanisms  by  which  it  might  come  to  be  and  strategies  by  which  it  might  be 
managed 

A  piece  of  software  or  firmware  known  as  the  monitor  resides  on  each  proces¬ 
sor  and  supports  the  basic  VIM  execution  environment  The  monitor  maintains  on 
each  processor  a  list  of  events  waiting  to  tie  executed  on  that  processor  This  list 
is  known  as  the  event  list 

I  et  us  assume  that  initially,  by  some  means  such  as  an  operator  typing  at  a 
keyboard,  one  processor  somewhere  in  the  network  has  been  given  an  event  to 
process  This  processor  will  now  have  one  event  on  its  event  list  The  goal  of  a 
processor  is  to  em^ty  its  event  list,  so  our  processor  will  take  the  new  event  off 
its  list  and  see  what  to  do  with  it  Most  likely,  the  event  will  cause  some  computa¬ 
tion  to  occur  and  then  result  in  a  new  event's  being  added  to  the  event  list, 
whereupon  the  whole  cycle  will  repeat  As  long  as  this  situation  persists,  and  each 
event  r  ouses  exactly  one  new  one  to  take  its  place,  there  is  little  opportunity  for 
other  processors  t  >  get  into  the  act 

let  us  suppose,  therefore,  that  at  some  point  an  event  causes  two  or  morn 


no 


Chapter  4  Architecture  of  Reference  Tree  Networks 


AD-A076  570  MASSACHUSETTS  INST  OF  TECH  CAMBRIDGE  LAB  FOR  COMPUTE— ETC  F/6  5/8 
REFERENCE  TREE  NETWORKS?  VIRTUAL  MACHINE  AND  IMPLEMENTATION. (U) 

JUL  79  R  H  HALSTEAD  N00014-75-C-0681 

UNCLASSIFIED  MIT/LCS/TR-222 _  NL 

2  or  3  I 

AD- 

4076570 


L 


HATlONAi.  BUREAU  0*  STANDARDS 

IBCJKKO**  •ffOU/TlO*  'ft*  'tUftf 


events  to  be  added  to  the  event  list.  Henceforth  there  will  be  several  events 


vying  for  our  processor's  attention  at  the  same  time  The  processor  might  continue 
to  operate  on  all  the  events  Itself  by  always  taking  the  oldest  event  from  the 
event  list,  producing  its  consequent  events,  and  placing  these  at  the  end  of  the 
event  list  Things  could  be  speeded  up,  though,  by  taking  advantage  of  the  other 
processors  in  the  system  If  there  are  enough  events  on  the  event  list,  the  over¬ 
head  of  sending  some  of  them  to  a  neighbor  should  be  worth  the  increased  proc¬ 
essing  power  thus  brought  to  bear  on  the  problem 

The  process  by  which  it  is  decided  what  events  to  execute  where  Is  the  sub¬ 
ject  of  subsequent  chapters,  before  taking  that  up  it  is  a  good  idea  to  have  a 
short  look  at  the  mechanics  of  moving  events  (and.  as  a  consequence,  objects) 
around  Sending  an  event  to  another  processor  is  simple  enough  all  that  is 
required  is  to  send  a  list  of  references  to  the  objects  participating  In  the  event 
This,  however,  represents  only  a  small  part  of  the  overhead  required  to  really  bring 
that  processor  into  the  action  Before  the  new  processor  can  make  any  sense  out 
of  the  event,  it  will  need  a  copy  of  the  text  of  the  target  object  of  the  event  If 
it  does  not  happen  to  already  have  this  text,  it  will  have  to  send  an  inquiry  for  it 
to  some  processor  which  does  *  While  this  inquiry  is  being  sent  and  replied  to,  the 
event  cannot  be  processed  Depending  on  the  nature  of  the  target  object,  further 
inquiry-response  cycles  may  he  required  to  gather  enough  information  to  process 
the  event  T  maliy,  when  the  event  is  processed,  a  new  event  or  events  will  most 
likely  be  generated,  containing  references  related  to  those  present  in  the  original 

*  T his  will  probably  be  the  original  sender  of  the  event,  however,  it  may  be  that 
no  neighbor  of  the  inquirer  has  a  copy  of  the  text  which  enables  if  to  reply  immedi¬ 
ately  to  the  inquiry.  In  this  case,  the  inquiry  must  be  forwarded  until  it  reaches  a 
processor  capable  of  replying,  whereupon  the  reply  must  bo  forwarded  back  to  the 
original  inquirer 
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frequently,  the  texts  of  these  objects  will  not  be  available  locally  either,  so  addi¬ 


tional  inquiry-response  delays  may  ensue  as  subsequent  related  events  are  proc¬ 
essed.  Thus  the  true  overhead  associated  with  sending  an  event  to  another  proc¬ 
essor  is  the  overhead  required  to  establish  a  "working  set"  for  that  event  and  its 
consequents  on  that  processor. 

Having  exposed  the  drawbacks  of  sending  an  event  to  another  processor.  It  Is 
worth  mentioning  some  mitigating  factors  first,  If  the  event  really  represents  the 
beginning  of  a  computation  that  will  proceed  for  a  time  without  requiring  too  much 
communication  with  other  activities,  using  another  processor  can  still  come  very 
close  to  doubling  the  effective  computing  speed  of  the  system  Second,  the 
efficiency  of  the  system  can  probably  be  improved  greatly  through  various  forms  of 
tuning  for  example,  the  sender  of  an  event  might  also  send  a  selection  of  object 
texts  likely  to  be  asked  for  by  the  receiver  in  the  process  of  setting  up  its  "work¬ 
ing  set  "  Similarly,  the  reply  to  an  inquiry  might  not  be  just  one  object  text,  but  a 
collection  of  related  texts  including  the  one  asked  for  These  strategies,  which 
are  analogous  to  virtual  memory  strategies  that  attempt  to  predict  which  pages 
should  be  kept  in  core,  might  not  do  much  to  reduce  the  number  of  bytes  sent  over 
communication  linns,  but  would  reduce  the  number  of  wasteful  delays  between 
sending  of  inquiries  and  receipt  of  responses 

Fortunately,  the  working  set  that  must  be  accumulated  by  an  event  running  on 
a  new  processor  can  often  be  reasonably  compact.  Suppose,  for  instance,  that  the 
event  is  a  function  receiving  arguments  and  a  continuation  A  working  set  that  will 
allow  this  commutation  to  proceed  for  a  while  would  include  the  body  of  the  func¬ 
tion  (or  at  least  those  parts  of  the  body  which  will  be  exercised  by  the  arguments 
given)  and  the  texts  of  the  arguments.  If  the  arguments  are  numbers,  no  extra 
Information  is  required  if  the  arguments  are  highly  structured,  it  is  probable 
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(depending  on  the  nature  of  the  function)  that  only  a  portion  of  the  texts  accessi¬ 
ble  from  the  argument  references  will  ever  be  accessed  Note  that  the  text  of  the 
continuation  is  not  needed  at  all  at  least  not  until  the  function  returns  a  value, 
which  might  be  a  natural  point  to  transfer  responsibility  back  to  the  first  processor, 
where  the  text  of  the  continuation  Is  presumably  located. 

To  summarise,  then,  we  can  imagine  the  system  beginning  to  operate  with  one 
event  at  a  particular  processor  Activity  will  spread  to  other  processors  as  the 
parallelism  of  the  program  increases  and  overloaded  processors  send  events  to 
their  less-loaded  neighbors  Perhaps  at  some  later  point  the  various  threads  of  the 
computation  will  either  die  out  or  begin  to  come  together  again  Thus  the  number 
of  active  processors  might  shrink  until  there  is  again  only  one  active  processor 
with  one  event  to  process  perhaps  "print  out  the  answer  "  Of  course,  another 
possibility  is  that  the  system  is  constantly  receiving  new  events  to  process,  as 
users  come  up  with  new  tasks  for  the  system  to  perform  This  will  lead  to  more  of 
a  steady-state  situation  where,  at  any  time,  some  tasks  are  just  beginning,  some 
just  ending,  and  others  in  various  growing  or  shrinking  phases.  In  either  case,  the 
system  will  suffer  from  some  amount  of  overhead  and  inefficiency  as  processors 
build  up  working  sets  for  newly  acquired  events,  but  may  gain  even  more  from  the 
application  of  extra  processing  power  to  its  business 
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We  now  consider  the  details  of  an  implementation  capable  of  supporting  the 
VIM  object  structure  these  details  fall  into  two  categories  those  involving  the 
bookkeeping  that  keeps  track  of  the  state  of  objects  and  their  whereabouts,  and 
those  pertaining  to  the  garbage-collection  of  objects. 

Keeping  track  of  an  object  in  a  reference  tree  network  is  accomjdished  by 
maintaining  the  processors  which  contain  references  to  the  object  In  a  connecferf, 
acyclic  graph  called  the  reference  free  for  that  object  Each  reference  tree  con¬ 
sists  of  some  subset  of  the  nodes  and  arcs  (processors  and  inter-neighbor  links)  of 
the  network  The  nodes  which  belong  to  the  reference  tree  are  chosen  to  include 
all  the  processors  in  the  network  which  contain  references  to  the  object,"  and  the 
arcs  are  chosen  in  such  a  way  that  (1)  the  reference  tree  is  connected,  i.e.,  it  is 
possible  to  reach  any  node  in  the  tree  from  any  other  node,  traveling  only  over 
arcs  that  are  in  the  tree,  and  (?)  the  tree  is  acyclic  in  that  the  arcs  in  the  tree 
form  no  closed  kxips  (Note  that  the  arcs  in  the  reference  tree  are  undirected, 
hence  requirement  (?)  means  that  there  should  be  no  undirected  cycles  )  Put 
another  way,  there  should  he  a  unique  path  (using  only  arcs  in  the  tree)  from  anv 
node  in  a  reference  tree  to  any  other  Additionally,  every  arc  in  a  reference  tree 
must  go  between  two  nodes  that  are  in  the  tree  Reference  trees  are  so  named 
because  they  form  unrooted  trees  embedded  in  the  network  (see  figure  6  1). 

It  is  important  to  note  that,  in  general,  the  reference  trees  for  different 
objects  need  bear  no  relation  to  each  other  In  particular,'  it  is  not  the  case  that 

"Keeping  a  reference  tree  for  an  object  connected  may  require  it  to  include 
certain  processors  which  do  not  contain  any  references  to  the  object  and  would 
not  otherwise  need  to  be  in  the  reference  tree  We  shall  return  to  this  problem 
later 
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Figure  5.1:  Fxamples  of  reference  trees  (In  heavy  lines) 


there  is  one  "reference  tree”  in  the  network,  used  for  all  objects.  Central  to  the 
concept  of  reference  trees  is  that  they  are  free  to  grow  and  shrink  dynamically, 
following  changes  in  the  roles  of  the  corresponding  objects  in  the  operation  of  the 
system. 

Also  significant  is  the  fact  that  reference  trees  can  be  maintained  by  a  com¬ 
pletely  distributed  mechanism  in  which  each  processor  In  a  tree  remembers  only  the 
state  of  its  immediately  adjacent  links  (/.«.,  whether  each  link  is  in  the  tree  01  not). 
Processors  not  in  the  tree  for  an  object,  of  course,  have  no  references  to  the 
object,  and  need  remember  no  information  about  it.  Even  the  cycle-free  nature  of 
the  tree  can  be  preserved  on  the  same  strictly  distributed  basis  —  no  central  clear¬ 
inghouse  is  needed  to  determine  whether  a  cycle  is  being  formed 
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Some  subset  of  the  processors  having  references  to  an  object  will  also  be 
custodians  of  a  copy  of  the  text  of  the  object.  The  management  of  object  texts 
has  three  basic  goals:  ( 1 )  to  ensure  that  no  object  text  is  "lost"  (i.e.,  to  ensure 
that  at  least  one  processor  has  custody  of  an  object's  text  at  all  times),  (2)  to 
keep  each  processor  in  the  reference  tree  for  an  object  apprised  of  the  directions 
In  which  it  may  send  inquiries  requesting  a  copy  of  the  text,  and  (3)  to  provide  a 
mechanism  for  performing  side  effects  on  object  texts  The  primary  reason  for 
requiring  reference  trees  to  be  connected  is  so  that  any  processor  with  a  refer¬ 
ence  to  an  object  can  always  communicate  with  all  custodians  for  that  object  sim¬ 
ply  by  following  links  that  are  part  of  the  object's  reference  tree  Indeed,  we  will 
require  that  all  communication  concerning  an  obit rc.t  travel  strictly  over  links  that 
are  in  the  reference  free  lor  that  object. 

The  acyclic  nature  of  reference  trees  guarantees  that,  once  a  message  has 
been  sent  across  a  particular  link,  it  can  never  "loop  back"  to  its  sender  unless 
either  ( 1  )  it  leaves  the  reference  tree,  or  (?)  some  processor  returns  the  message 
back  along  the  same  link  over  which  it  was  received  Thus  when  a  processor 
wants  to  send  a  message  to  a  custodian  of  a  text,  all  it  needs  to  know  locally  is 
"which  way"  to  send  it  initially,  which  of  its  neighbor.,  in  the  reference  tree 

lies  along  the  unique  path  from  it  to  the  desired  custodian  It  does  not  even  need 
to  know  the  identity  of  the  custodian  since  the  message  can  be  forwarded  from 
node  to  node  using  only  the  local  information  telling  which  way  the  desired  custo¬ 
dian  lies  Consequently,  the  only  information  processors  need  to  record  concerning 
the  location  of  custodians  is  which  ways  (through  the  reference  tree)  they  may  be 
found  This  in  turn  means  that  local  changes  in  custody  need  only  cause  data  base 
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updates  In  the  processors  directly  involved  —  the  remainder  of  the  reference  tree 


will  not  be  affected. 

gtext  requests  may  be  handled  using  an  inquiry-response  protocol  in  which  a 
processor  containing  an  event  which  has  performed  a  gtext  request  for  an  object 
may  send  an  inquiry  through  that  object's  reference  tree  to  a  custodian  The  cus¬ 
todian  may  then  reply  by  sending  a  copy  of  the  text  of  the  object  back  to  the  ori¬ 
ginal  inquirer 

Since  a  gtext  request  is  for  read-only  access,  multiple  copies  of  an  object 
text  may  simultaneously  serve  gtext  requests  in  different  locations  The  handling 
of  locktext  requests  is  more  complicated  The  basic  strategy  for  satisfying  a 
locktext  request  is  to  eliminate  all  redundant  copies  of  the  text  in  question,  leav¬ 
ing  only  one  copy  at  the  site  where  the  locktext  request  was  made  Once  this 
has  been  accomplished,  thp  requestor  may  perform  any  desired  side  effects  on  the 
text,  without  fear  that  some  other  copy  of  the  text  might  escape  the  update 
Unfortunately,  this  strategy  leads  to  new  possibilities  for  deadlock:  if  two  proces¬ 
sors  simultaneously  try  to  perform  locktext  requests  for  the  same  object  (each 
requesting  to  have  the  unique  remaining  copy  of  the  object’s  text),  some  means 
must  exist  for  inducing  all  the  processors  in  the  object's  reference  tree  to  co¬ 
operate  in  satisfying  first  one  of  the  requests  and  then  the  other.  The  way  this  is 
accomplished  in  a  reference  tree  network  is  by  associating  with  each  locktext 
request  a  unique  priority  chosen  from  some  totally  ordered  set.  Then  at  every 
processor  which  must  co-operate  in  satisfying  a  locktext  request,  priorities  of 
conflicting  requests  can  be  compared  and  the  highest-priority  request  honored  Fur¬ 
thermore,  since  gtext  requests  can  conflict  with  locktext  requests,  gtext-type 
Inquiries  must  also  carry  priorities. 

A  minor  augmentation  of  the  scheme  suggested  in  the  preceding  caragraph  is 
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necessary  to  avoid  the  problem  of  cyclic  res/arf[32].  An  example  of  cyclic  restart 
may  occur  in  the  presence  of  two  events,  A  and  B,  and  two  objects,  X  and  Y , 
where  execution  of  A  first  requests  locktext  access  to  X  and  then  Y,  and  execu¬ 
tion  of  B  requests  locktext  access  to  Y  and  then  X  Suppose  A  is  executing  and 
has  acquired  access  to  X.  Then  B  begins  to  execute,  acquiring  access  to  y 
When  A  asks  for  access  to  Y,  its  execution  may  be  aborted  and  restarted,  once 
again  acquiring  access  to  X  When  B  requests  access  to  X,  B  may  be  aborted  and 
restarted,  leading  to  a  repetition  of  the  above  cycle.  This  kind  of  "dynamic 
deadlock"  may  be  avoided  by  establishing  a  total  priority  order  on  events,  rather 
than  just  Individual  requests  Then  if  event  A  has  a  higher  priority  than  event  B. 
any  access  privileges  ever  requested  during  execution  of  A.  even  during  execu¬ 
tions  that  were  subsequently  abored.  could  be  made  unavailable  to  B  until  success¬ 
ful  completion  of  the  execution  of  A  A  simple  measure  of  an  event's  priority,  and 
one  which  is  consistent  with  fair  scheduling,  is  its  "age"  (suitably  qualified,  as  with 
a  processor  ID.  to  make  it  unique)  Using  this  measure,  there  will  always  be  an 
"oldest"  event  which  will  be  able  to  accumulate  all  the  resources  it  needs  in  order 
to  complete  Then  the  second  oldest  event  will  be  able  to  complete,  and  so  on, 
avoiding  cyclic  restart 

Consequently,  we  use  time  stamps  as  priorities,  and  associate  higher  priority 
witt'  smaller  (older)  time  stamps  It  is  not  necessary  that  all  processors  have 
access  to  a  common  clock  in  order  to  generate  these  time  stamps,  a  method  by 
which  these  time  stamps  can  be  obtained,  and  their  uniqueness  guaranteed,  is  dis¬ 
cussed  below  In  section  b  i  d 

We  now  discuss  in  more  detail  the  management  of  object  texts.  For  each  out¬ 
going  link  of  each  reference  tree  it  belongs  to,  a  processor  maintains  the  informa- 
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tion  shown  in  Table  5.2. 


•  "text-this-way"  bit 

•  "request-received"  held  (values  are  NULL,  INQUIRY,  and  tOCK.) 

•  request  time  stamp 

Table  5.2:  Text  management  information 

for  each  reference  tree  it  belongs  to,  a  processor  also  records  whether  it  has  a 
copy  of  the  text  of  the  corresponding  objoct  (and,  if  so,  the  memory  address  of 
the  copy1).  Additionally,  it  keeps  track  of  all  events  on  that  processor  currently 
waiting  for  or  using  the  text,  and  the  timestamps  of  those  events. 

for  reasons  discussed  below,  every  reference  tree  link  has  an  asymmetry,  the 
processors  at  Its  ends  bearing  a  master-slave  relationship  to  each  other  In  a 
stable  condition,  one  end  of  a  link  (for  a  particular  reference  tree)  is  always  mas¬ 
ter,  and  the  other,  slave  Transiently,  both  may  be  slaves,  but  both  ends  of  a  link 
can  never  simultaneously  be  masters  of  that  link  The  way  in  which  the  master- 
slave  relationship  is  recorded  and  manipulated  is  described  in  a  subsequent  sec¬ 
tion  For  now,  suffice  It  to  say  that  it  is  possible  for  a  processor  to  determine 
whether  It  is  master  of  a  particular  link  of  a  particular  reference  tree;  that  a  proc¬ 
essor  can  relinquish  its  mastery  of  a  link  to  the  processor  at  the  other  end  by 
sending  a  suitable  message;  and  that  a  processor  that  is  not  master  can  send  a 
"mastery-request"  message  which  will  induce  the  master  to  relinquish  Its  mastery 
of  the  link  The  pieces  of  information  enumerated  in  Table  5  2,  plus  mastery  infor- 

*Various  encoding  schemes  can  be  used  to  reduce  the  average  amount  of 
memory  needed  to  store  this  information.  For  example,  the  contents  of  the  request 
time  stamp  will  only  be  relevant  If  the  "request-received"  fleld  Is  not  NULL,  presum¬ 
ably  an  infrequent  occurrence,  on  the  average,  for  any  particular  object. 
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mation,  constitute  all  the  data  used  in  manipulating  object  texts. 


5.1.1:  Adjustments  to  Custody 

Transmission  of  object  texts  between  processors  will  frequently  occur  in 
response  to  inquiries  or  other  such  stimuli,  but  can  also  occur  "spontaneously"  as  a 
result  of  garbage-collection  activity  or  storage-load  balancing  among  processors. 
The  spontaneous  case  is  simpler  to  discuss,  and  illustrates  mechanisms  common  to 
all  communications  of  texts  from  one  processor  to  another  We  deal  only  with 
movement  of  copies  of  an  object’s  text  among  processors  already  in  the  reference 
tree  for  the  object;  expansion  and  contraction  of  reference  trees  will  be  dis¬ 
cussed  lator 

The  representation  of  texts  sent  over  communication  links  cannot  be  discussed 
without  reference  to  concepts  not  yet  Introduced;  for  now,  we  note  simply  that  a 
suitable  representation  exists 

Explaining  object  text  management  involves  discussing  the  meaning  of  each 
text  management  datum  in  Table  5  ?,  and  the  mechanisms  by  which  these  data  are 
used  and  updated  The  "text-this-way"  bit  for  a  link  indicates  whether  a  custodian 
of  the  text  can  be  reached,  through  the  reference  tree,  starting  with  the  link  (see 
Figure  53)  The  "text-this-way"  bit  need  be  updated  only  when  custody  of  a  text 
changes 

The  simplest  kind  of  custody  change  occurs  when  a  processor  ceases  being  a 
custodian  by  simply  deleting  its  copy  of  a  text  Tor  example,  in  Figure  5.3(a),  two 
processors  have  copies  of  the  object  text  It  should  be  possible  for  one  of  them 
to  delete  its  copy  without  any  damage  being  done  The  problem  is  that  processors 
In  a  reference  tree  must  somehow  agree  on  which  copy  of  the  text  will  be  kept,  a 
negotiation  possibly  global  in  scope  Simply  knowing  that  another  custodian  exists 
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(a)  (b)  (c)  (d) 


Heavy  lines  indicate  reference  tree  links ,  shaded  boxes  indicate 
processors  with  copies  of  the  tent,  "lext-thi s-way"  bit  is  shown  next 
to  each  reference  tree  link  emanating  from  a  processor . 

Figure  5  3  "  I  ext-this-way"  bits 

is  not  enough  to  allow  a  processor  to  delete  its  copy  of  a  text,  for  the  other  cus¬ 
todian  could  then,  symmetrically,  make  the  same  decision,  and  all  copies  of  the 
object  text  would  be  lost 

Negotiation  can  be  avoided  If  every  processor,  just  before  deleting  Its  copy  of 
a  text,  sends  the  text  to  another  processor,  thus  ensuring  that  the  information  will 
not  be  lost  If  the  text  message  arrives  at  another  processor  already  in  posses¬ 
sion  of  a  copy  of  the  text,  the  message  can  be  discarded.  Otherwise,  the  recipi¬ 
ent  processor  must  become  a  custodian  for  the  text  (although  it  always  has  the 
option  of  passing  it  on  to  yet  another  processor)  If  the  two  custodians  in  Figure 
b  3(a)  both  transfer  their  custody  to  the  center  processor,  one  text  message  will 
be  received  and  the  other  discarded,  leading  to  the  situation  in  f  igure  5.3(b). 

Note  that  every  reference  tree  has  embedded  in  it  a  (possibly  null)  subtree 
which  includes  all  the  processors  that  can  reach  a  text  in  more  than  one  direction, 
l.e.,  that  have  at  I  "»**  two  "text-thls-way"  bits  set  In  Figure  5.3(d),  this  subtree 
consists  of  the  top  center  and  middle  center  processors.  Whenever  such  a 
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subtree  exists,  it  is  bounded  by  two  or  more  custodians  (upper  left,  upper  right, 
and  bottom  center,  in  figure  6  3(d))  which  each  have  only  one  "text-this-way"  bit 
set  As  long  as  each  of  these  custodians  obeys  the  text-preservation  discipline 
set  forth  above,  at  least  one  copy  of  the  text  is  guaranteed  to  be  preserved  no 
matter  what  the  processors  In  the  interior  subtree  do  with  any  copies  they  might 
have  Therefore,  we  may  exempt  these  interior  processors  from  the  requirement  of 
sending  out  a  copy  of  the  text  before  deleting  it  The  rule  used  by  the  processors 
is  if  at  least  (wo  "te*t-this-way"  bits  for  an  object  are  set,  a  local  copy  of  the 
objec  t's  fe*r  may  be  deleted  without  sending  any  messages  Note  that  such  a  dele¬ 
tion  (as  by  the  top  (enter  processor  in  figure  6  3(d))  does  not  require  adjusting 
any  "text- this- way "  bits 

In  general,  sending  a  text  from  one  processor  to  another  requires  adjusting 
some  "text-this-way"  bits  However,  it  only  requires  ad  just  i  ng  the  bits  pertaining 
to  the  link  over  which  the  te»t  was  sent  1  he  sending  processor  always  sets  its 
“text-this-way"  bit  for  the  link,  recording  the  fact  that  the  text  is  out  in  that  direc¬ 
tion,  and  will  continue  to  be,  at  least  until  a  copy  of  the  text  comes  back  over  the 
same  link  The  receiving  processor,  however,  must  be  told  by  the  sender  how  to 
set  its  "text-this-way"  bit  for  the  link  Presumably  the  bit  is  a  1  before  the  mes¬ 
sage  is  received  (else  the  receiver  would  be  very  surprised  to  sen  a  text  coming 
from  that  direction1)  The  question,  then,  is  whether  there  will  continue  to  be  a 
text  in  that  direction  after  the  message  If  the  sender  keeps  its  copy  of  the  text, 
the  answer  is  yes  If  there  is  another  custodian  in  the  same  direction  from  the 
receiver  as  the  sender  ("behind"  the  sender,  as  it  were)  then  the  answer  is  yes 
If  there  is  no  such  custodian  and  the  sender  is  going  to  delete  its  copy  of  the  text 
upon  sending  the  message,  then  the  Answer  is  no  Corresponding  to  these  alterna¬ 
tives  are  two  kinds  of  text  messages,  T*  if  the  receiver  should  set  Iris  "text-this- 
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way"  bit,  anil  !  it  bo  should  cloar  it  For  example,  going  from  figure  f)  3(a)  or  (c) 
to  (b).  the  center  processor  would  receive  T  messages;  going  from  (b)  to  (a)  or 
(c),  It  would  send  T*  messages 


1 


T- 


(■)  (b)  (c) 

Figure  5  4  Race  condition  in  sending  texts 

Unfortunately,  this  updating  scheme  can  load  to  an  inconsistency  among  the 
"text-this-way"  bits  Figure  5.4  shows  what  would  happen  if  two  neighboring  proc¬ 
essors  simultaneously  decided  to  delete  their  copies  of  a  text  f  ach  would  be 
required  to  send  a  T  message  to  the  other  just  before  deleting  its  copy  I  igure 
5  4(b)  shows  the  two  T  messages  in  transit  Upon  receiving  its  message,  each 
processor  would  obediently  clear  its  "text-this-way"  bit  and  once  again  become  a 
custodian  of  the  text,  leading  to  the  inconsistent  state  in  figure  5  4(c)  This  can 
be  prevented  if  a  processor  is  prohibited  from  sending  a  text  message  over  a  link 
that  it  is  not  master  of  Since  both  ends  of  a  link  cannot  simultaneously  be  master, 
the  situation  depicted  in  Figure  5  4  can  never  arise  No  unsolvable  problems  are 
created  by  this  restriction  a  processor  desiring  to  send  a  text  over  a  link  of 
which  it  is  not  the  master  must  simply  send  a  "mastery-request"  message  and 
await  the  reply 
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5.1.2:  Inquiry  Processing 

We  now  turn  to  the  handling  of  inquiries  from  processors  wanting  to  obtain 
copies  of  a  text.  Such  inquiries  will  result  from  gtext  or  locktext  requests  made 
by  events  executing  on  those  processors  for  the  time  being,  we  restrict  our 
attention  to  inquiries  of  the  gtext  variety 

When  a  gtext  request  is  issued  for  a  text  not  available  locally,  the  processor 
on  which  the  request  is  issued  has  two  alternatives  it  can  send  ttie  requesting 
event  to  mother  processor  whore  its  chances  of  satisfying  the  request  will  be 
better,  or  it  can  send  out  an  inquiry  to  bring  in  the  requested  text  Assuming  It 
chooses  the  latter,  the  processor  must  select  a  neighbor  for  whom  the  "text-this 
way"  bit  is  sot  and  send  an  inquiry  message  to  that  neighbor  (If  the  inquiring 
processor  is  currently  master  of  the  link,  it  would  do  well  to  first  relinquish  mastery 
to  its  neighbor,  facilitating  the  return  of  the  desired  text  ) 

When  a  proc  essor  re>  eives  an  inquiry  message  it  responds  immediately  it  it  is 
able  (r  e.,  it  has  a  copy  of  the  text,  is  master  of  the  link,  and  no  pending  locktext 
request  with  priority  prevents  it  from  sending  out  the  requested  text)  If  it  is  not 
able  to  reply  immediately,  it  sets  the  "request  received"  held  for  the  link  over 
which  the  inquiry  was  received  to  have  the  value  INQUIRY  and  records  the  time 
stamp  of  the  inquiry  in  'no  "request  time  stamp"  held 

If  an  inquiry  cannot  be  replied  to  because  the  replier  is  not  master  of  the 
relevant  link,  a  "mastery-request"  message  must  be  sent  When  mastery  has  been 
obtained,  the  reply  may  then  be  sent  (assuming  no  other  circumstance  then 
prevents  It)  If  an  immediate  reply  cannot  be  made  due  to  a  locktext  request  with 
priority,  no  further  action  need  be  taken  When  the  event  requesting  the  locktaxt 
finishes,  the  inquiry  will  be  honored 

If  a  processor  receives  an  Inquiry  for  a  text  which  it  does  not  have,  then  it 
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will  in  general  have  to  forward  the  inquiry.  1  he  "request-received"  held  set  upon 
rocoiving  the  inquiry  records  the  direction  in  which  the  desired  text  should  be 
returned  when  it  is  finally  obtained 

forwarding  of  Inquiries  contains  a  couple  of  pitfalls  to  avoid  Generally  speak¬ 
ing,  forwarding  of  an  inquiry  received  is  analogous  to  sending  of  an  Inquiry  needed 
to  satisfy  some  Internally  generated  need  In  either  case,  some  neighbor  In  whose 
direction  a  text  exists  Is  solected  as  the  target  of  the  inquiry.  However,  a  neigh¬ 
bor  selected  for  forwarding  should  not  bo  the  same  neighbor  from  whom  the  original 
inquiry  was  recolved,  or  else  an  Infinite  loop  of  forwarding  inquiries  could  occur  in 
the  situation  shown  in  Figure  ft  ft 


Proper  forwarding 

"  *  Inquiry^ 

■  - □ - □ - ■ 

ypx,  Improper  Forwarding  Text 


Figure  ft  ft  Inquiry  received  from  direction  of  text 


It  is  possible  for  there  to  he  no  legitimate  direction  for  forwarding  an  Inquiry  (If 
the  only  direction  In  which  a  text  can  be  found  is  in  the  direction  of  the  sender  of 
the  original  inquiry)  However,  the  sender  must  have  thought  there  was  a  text  in 
the  direction  it  chose  for  sending  the  original  Inquiry  Therefore,  unless  the  refer¬ 
ence  tree  data  base  has  broken  down,  the  inquiry  must  have  "crossed  in  the  mail" 
a  T  message  to  the  inquirer  containing  the  desired  text  (see  Figure  ft  R)  In  other 
words,  the  Inquiry  barf  a/ready  been  rep/ied  to.  even  before  it  was  received  In 
such  a  case,  the  Incoming  Inquiry  message  can  simply  he  ignored 

When  a  processor  that  has  "request-received"  fluids  set  to  INQUIRY  receives  a 
copy  of  a  desired  text.  It  should  send  a  copy  of  the  text  over  every  link  over 
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Inquiry 


|—  j  inquiry  ^ 

_ 

T-  Message 


lext 


Figure  fi.R  Inquiry  crossing  text  message 


which  an  inquiry  for  that  text  was  recelvod,  clearing  each  •'request-received'1  field 
to  NLJl  L  as  the  corresponding  text  message  Is  sent 


6.1.3:  Side  Effect  Management 

When  an  executing  event  issues  a  locktext  request,  and  It  is  decided  to  con¬ 
tinue  execution  of  the  event  on  the  same  processor,  steps  must  be  taken  to 
ensure  that  a  copy  of  the  relevant  text  is  brought  to  that  processor,  and  that  all 
othor  copies  are  deleted.  These  conditions  are  satisfied  if  and  only  if  the  proces¬ 
sor  has  a  copy  of  the  object's  text  and  all  the  processor's  "text-this-way"  bits  tor 
the  object  are  zero.  If  some  of  the  "text-this-way"  bits  are  nonzero,  then  IOCK 
messages  must  be  sent  out  over  those  links  connected  to  the  processor  that  have 
nonzero  "text-this-way"  bits 

A  LOCK  message  Is  a  request  for  the  processor  receiving  it.  and  all  other  proc¬ 
essors  in  its  part  of  the  reference  tree  (all  processors  leachable  from  it  through 
the  reference  tree  without  using  the  link  over  which  the  IOCK  message  was 
received),  to  delete  their  copies  of  the  relevant  object  text  Thus  a  processor 
receiving  a  IOCK  message  will,  in  general,  set  the  corresponding  "request- 
received"  held  to  IOCK  and  forward  the  IOCK  message  over  every  link  whose 
"text-this-way"  bit  Is  set,  except  for  the  link  over  which  the  original  LOCK  message 
was  rece  ved  A  processor  that  roceives  a  IOCK  message  but  has  nowhere  to  for- 
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ward  it  to  Is  expected  to  delete  its  copy  of  the  text  and  respond  with  a  1  mes¬ 
sage  as  an  Indication  of  its  compliance  ^  A  processor  that  has  forwarded  some 
LOCK,  messages  is  expected  to  wait  until  a  1  message  has  been  received  in 
response  to  each,  then  delete  its  own  copy  of  the  text  and  make  Its  own 
response,  In  the  form  of  a  T  message  sent  over  the  link  whose  "request-received'1 
field  had  been  set  to  IOCK  finally,  when  the  original  processor  on  which  the  lock- 
text  was  requested  has  received  a  T  message  'rom  each  neighbor  that  had  a 
text,  it  can  proceed  with  execution  of  the  requesting  event 


5.1.4:  Conflict  Resolution 

What  has  Just  been  described  Is  the  simple  case,  where  only  a  single  locktext 
request  is  pending  in  the  whole  reference  tree  In  an  actual  situation,  a  processor 
may  receive  l  OOK.  messages  for  the  same  object,  originating  on  different  proces¬ 
sors,  over  several  different  links  To  further  complicate  matters,  it  could  simultane¬ 
ously  receive  inquiry  messages  for  the  same  object  over  yet  other  links  Finally, 
the  processor  could  have  locally  executing  events  also  requesting  various  kinds  of 
access  to  the  object  A  processor  In  such  a  situation  must  serve  as  an  arbiter  and 
determine  which  of  a  set  of  conflicting  requests  to  honor  first 

To  handle  these  conflicts,  I  OOK  messages,  inquiry  messages,  and  events  them¬ 
selves,  carry  time  stamps*  tvery  event  Is  assigned  a  unique  time  stamp  when  it 

^lt  is  possible  to  devise  protocols  that  do  not  require  actual  text  messages  to 
be  sent  back  from  all  the  leaves  in  response  to  lOCk  messages  A  short  "IOCK 
acknowledge"  message  can  be  sent  instead  Unfortunately,  such  a  scheme  compli¬ 
cates  the  handling  of  inquiries  by  invalidating  the  promise  illustrated  In  Tigure  5  6. 

*On  the  MuNot,  inquiries  are  not  time-stamped  This  is  adequate  in  every  case 
except  that  of  an  objoct  which  Is  named  in  gtext  requests  and  also  continually  in 
locktext  requests  Under  these  circumstances,  the  locktext  requests  will  take 
precedence  and  the  gtext  requests  may  never  be  satisfied  This  has  not  occurred 
to  any  observable  degree  in  normal  operation  but  must  be  regarded  as  a  "hole"  in 
the  MuNet  implementation 
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is  created  ^  In  principle,  the  precise  algorithm  for  assigning  time  stamps  to  events 
is  unimportant,  so  long  as  it  never  assigns  the  same  time  stamp  to  more  than  one 
event  However,  in  practice,  it  is  desirable  if  the  time  stamps  assigned  increase 
with  time,  so  that  priority,  which  accrues  to  the  events  bearing  the  lowest  time 
stamps,  will  accrue  to  the  events  that  have  been  waiting  the  longest  It  is  not 
necossary  to  use  a  globally  accessible  clock  to  achieve  this,  a  set  of  local  clocks 
will  do,  provided  that  they  are  kept  sufficiently  synchronized  that  fairness  criteria 
are  not  grossly  violated 

On  the  MuNet.  oach  local  clock  is  a  counter  which  Is  incremented  once  before 
each  use  I  his  provision  is  sufficient  to  guarantee  that  no  single  processor  will 
ever  give  the  same  time  stamp  to  more  than  one  event.  To  assure  that  processors 
acting  independently  will  not  give  out  the  same  time  stamps,  each  processor 
appends  its  unique  processor  II)  to  the  value  read  out  from  its  counter  Whenever 
a  request  or  event  message  bearing  a  time  stamp  is  received  at  a  processor,  the 
processor's  time  stamp  counter  is  increased,  if  necessary,  to  match  the  time  stamp 
value  in  'he  message  To  make  sure  that  rough  synchrony  is  maintained  even  in 
the  absence  of  request  or  event  traffic,  each  processor  periodically  sends  its 
current  time  stamp  value  to  each  of  its  neighbors  ^ 

The  need  to  have  time  stamps  for  conflict  resolution  is  unfortunate,  since  it 
imposes  a  space  requirement  of  unknown  size,  depending  on  how  long  the  system  is 
expected  to  operate  Perhaps  some  scheme  can  be  devised  for  "garbage- 
collocting"  and  recycling  time  stamps  that  have  fallen  into  disuse,  but  any  such 

Actually,  on  the  MuNet,  an  event  only  receives  a  time  stamp  when  it  first 
needs  one  for  purposes  of  these  protocols  This  policy  reduces  the  speed  with 
which  time  stamps  are  used  up,  but  does  not  introduce  any  indetermmacies  Into  the 
processing  of  requests 

^for  a  further  discussion  of  synchronization  of  clocks  in  distributed  systems, 
see  Lamport[22] 
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schem«  must  preserve  the  ordering  among  all  entities  currently  bearing  time 
stamps  No  such  scheme  was  considered  during  the  course  of  this  research 

A  processor  needing  to  arbitrate  among  a  set  of  requests  uses  their  time 
stamps  to  establish  a  priority  order  among  them  It  then  acts  to  satisfy  as  many 
as  possible,  subject  to  the  restriction  that  no  request  be  satisfied  at  the  expense 
of  a  request  with  an  earlier  time  stamp.  In  practice,  this  means  that  If  the  request 
with  the  earliest  time  stamp  is  a  locktext-type  request,  it  is  satisfied  first  and  all 
other  requests  are  held  in  abeyance  If  the  request  that  has  priority  is  a  gtext- 
type  inquiry,  it  and  all  gtaxts  up  to  the  earliest  locktext  can  be  honored  simultane¬ 
ously  * 

By  those  means,  a  processor  can  determine  which  of  the  various  requests  of 
which  it  is  aware  should  be  satisfied  first  It  does  not  follow  that  the  processor 
will  singlehandedly  have  the  wherewithal  to  satisfy  the  selected  request  For 
example,  if  tho  highest-prlority  request  is  an  inquiry,  it  cannot  be  satisfied  without 
possession  of  a  copy  of  the  relevant  text  Thus,  in  order  to  satisfy  the  request  It 
picks  as  the  most  urgent,  the  processor  may  need  to  forward  this  request  to  one 
or  more  of  its  neighbors  fach  of  those  neighbors  will  then  compare  the  priority  of 
the  forwarded  request  to  those  of  all  the  others  it  has  received,  and  again  act  on 
the  most  urgent  one,  holding  the  others  in  abeyance  The  only  external  information 
a  processor  needs  in  order  to  make  this  determination  is  the  type  (r.e.,  inquiry  or 
IOCK)  and  time  stamp  of  the  most  urgent  request  which  is  known  to  each  of  its 
neighbors  and  which  it  must  co-operate  in  satisfying  Thus  whenever  a  processor 

*lf  satisfaction  of  the  request  that  has  priority  also  incidentally  satisfies  some 
request  made  by  an  event  with  a  much  later  time  stamp,  it  is  permissible  to 
attempt  to  execute  this  latter  event  while  the  former  request  Is  still  active.  What 
Is  not  permissible  is  to  take  actions  to  satisfy  the  latter  request  which  would  be 
inconsistent  with  the  treatment  required  to  satisfy  the  former 
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sends  an  Inquiry  or  IOCK  message  to  a  neighbor,  it  is  in  effect  saying,  "lhis  is  now 
my  most  urgent  request  involving  this  object.  Its  urgency  Is  Indicated  by  the  time 
stamp  in  this  message  You  may  forget  any  previous  requests  I  have  sent  you 
regarding  this  object,  for  I  will  resend  any  of  them  if  and  when  it  becomes  my  most 
urgent  request  and  your  co-operation  is  still  needed  ” 

A  strategy  that  reconciles  our  jirevious  discussion  of  text  management  with 
this  philosophy  is  as  follows  when  an  inquiry  or  LOCK  message  arrives  at  a  proc¬ 
essor,  set  the  "request-received"  field  accordingly  and  record  the  time  stamp  from 
the  message  in  the  "request  time  stamp"  field  tor  that  object  for  that  link  (see 
table  5  2)  'f  some  other  request  for  the  same  object  active  on  the  same  proces¬ 
sor  has  a  lower  time  stamp,  take  no  further  action  the  newly  received  request 
must  wait  until  it  has  top  priority  +  If  the  newly  received  request  has  the  lowest 
time  stamp  of  all,  however,  It  must  be  honored  immediately,  using  the  algorithms 
described  above  for  handling  inquiries  and  LOCK  messages  If  those  algorithms  call 
for  forwarding  of  requests,  the  forwarded  requests  must  bear  the  same  time  stamp 
as  the  original 

An  inquiry  can  be  considered  satisfied  when  a  text  (either  7  ♦  or  T  )  message 
Is  sent  back  over  the  same  link  from  which  the  inquiry  was  received.  Therefore, 
when  a  T ♦  or  I  message  is  sent  over  a  link  whose  "request-received"  field  is  set 
to  INQUIRY,  the  "request-received"  field  ran  be  reset  to  NULL.  This  can  be  done 
oven  without  knowing  whether  the  text  message  was  intended  as  a  response  to 
the  inquiry  or  just  happened  to  l  e  sent  while  the  inquiry  was  active  Likewise,  a 
LOCK  request  can  bo  considered  satisfied,  and  the  corresponding  "request- 
received"  field  cleared  to  NUI I  .  when  a  T  message  is  sent  back  over  the 

^Except  that  If  the  request  is  a  gtext-style  Inquiry,  and  all  requests  with  lower 
time  stamps  are  also,  then  all  may  be  satisfied  simultaneously 
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requesting  link 


When  a  text  message  has  been  sent  over  a  link,  and  the  “request-received" 
held  cleared,  there  is  instantaneously  no  request  active  over  the  link,  this  situa¬ 
tion  will  continue  at  least  until  the  processor  at  the  other  end  of  the  link  has  itself 
disposed  of  the  request,  bringing  top  priority  (for  its  local  operations)  to  another 
request  for  the  object  then  if  the  processor  which  had  sent  the  text  must  once 
again  co-operate  in  satisfying  the  new  request,  It  will  receive  another  request  mes¬ 
sage 

Note  that  no  request  message  is  ever  sent  to  a  neighbor  whose  "text-this- 
way"  bit  is  not  set  thus  processors  at  the  fringes  of  a  reference  tree  will  only 
send  requests  inward,  toward  the  areas  of  the  reference  tree  where  texts  exist. 
Requests  will  never  be  sent  outward  past  the  last  text  in  any  particular  branch  of 
a  reference  tree  Consequently,  if  a  processor  has  sent  a  t  message,  in 
response  to  a  LOCK  request,  say.  it  will  thencoforth  receive  only  inward-directed 
requests,  but,  until  the  processor  once  again  has  a  copy  of  the  text,  none  from  the 
center  of  the  reference  tree  (defining  the  "center"  as  that  subtree  of  the  refer¬ 
ence  tree  which  contains  all  custodians  in  the  reference  tree  and  has  a  custodian 
at  every  leaf  node) 

Discussion  thus  far  has  centered  on  the  treatment  of  requests  received  from 
neighbors  Requests  arising  from  locally  executing  events  are  treated  in  a  very 
similar  fashion  If  processing  such  a  request  involves  sending  messages  to  neigh¬ 
bors,  the  time  stamp  of  the  requesting  event  is  used  in  those  messages  When  the 
conditions  of  the  request  are  met  (a  copy  of  the  text  available,  for  gtext; 
exclusive  access  to  the  text,  for  locktext),  execution  of  the  requesting  event  may 
be  resumed  This  resumption  may  conceivably  lead  to  other  requests,  for  other 
objects,  but  all  requests  made  by  an  event  rema;..  active  (r'.e.,  must  continue  to  be 
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satisfied)  until  execution  of  the  event  has  been  completed  successfully,  or  until 


the  event  is  aborted.  Therefore,  If  a  request  from  another  event  with  a  lower  time 
stamp,  either  on  the  same  processor  or  on  a  different  one,  conflicts  with  the  contin¬ 
ued  satisfaction  of  a  request  made  by  a  currently  suspended  event,  this  latter 
event  must  be  aborted  so  that  the  higher-priority  request  can  be  honored.  This 
does  not  cancel  the  requests  made  by  the  aborted  event;  they  continue  to  be 
active,  and  to  take  priority  except  where  temporarily  superseded  by  requests  with 
even  lower  time  stamps  Hut  satisfying  one  of  the  superseding  requests,  without 
aborting  an  event  whose  request  has  been  superseded,  violates  the  guarantees 
about  access  rights  made  by  the  definitions  of  gtext  and  locktext.  Therefore, 
such  an  event  must  be  aborted  and  held  for  re-execution  at  a  time  when  all  its 
requests  can  be  given  top  priority  * 

5.1.5:  Summary 

Within  the  framework  of  reference  trees,  an  object  text  management  protocol 
can  be  devised  which  is  capable  of  handling  multiple  copies  of  object  texts,  reduc¬ 
ing  the  number  of  copies  to  one  when  a  side  effect  Is  to  be  performed,  and  arbi¬ 
trating  between  asynchronously  generated  requests  of  differing  priorities.  Loca¬ 
tions  of  texts  within  a  reference  tree  are  recorded  by  means  of  a  distributed  set 
of  "text-this-way"  bits  kept  at  the  various  processors  in  the  reference  tree. 
These  bits  do  not  indicate  the  exact  locations  of  texts,  but  only  the  directions, 
traveling  through  the  reference  tree,  in  which  they  may  be  found  As  a  result,  local 
motions  of  texts  require  only  local  updates  of  "text-this-way"  bits;  other  proces- 

*On  the  MuNet,  some  complication  is  avoided  by  instantly  aborting  any  event 
whose  request  cannot  be  immediately  satisfied  Thus  no  "suspended"  events  ever 
exist,  only  aborted  events  awaiting  re-execution 


Section  6.1.6:  Summary 


119 


sors  in  the  reference  tree  do  not  need  to  be  informed 


Ihe  object-custody  protocol  allows  processors  to  request  read-only  (gtext)  or 
read/write  (locktext)  copies  of  object  texts  on  behalf  of  events  executing  on 
them  Each  such  request  must  be  labeled  with  the  priority  of  the  requesting  event, 
In  the  form  of  a  unique  time  stamp  These  time  stamps  are  used  to  resolve 
conflicts  between  requests,  and  the  "text  this-way"  bits  are  used  to  forward 
requests  to  all  processors  that  must  co-operate  in  satisfying  them 

By  recording  only  partial  information  about  text  locutions  (only  "text-this-way" 
bits  instead  of  the  actual  identities  of  custodians  of  texts),  this  object  text 
management  protocol  throws  away  the  opportunity  to  perform  certain  kinds  of 
optimizations,  such  as  sending  a  request  to  the  nearest  processor  with  a  copy  of  a 
desired  text  In  return,  updating  the  information  Is  very  simple  and  economical, 
encouraging  a  fluid  situation  in  which  texts  may  be  moved  freely  to  balance  loads 
or  adapt  to  changing  access  patterns  Moreover,  all  essential  operations  (determin¬ 
ing  whether  there  is  more  than  one  copy  of  a  text,  reaching  all  custodians  of  a 
text,  efc.)  can  still  bo  performed  in  a  straightforward  fashion 

6.2:  Garbage  Collection 

A  reference  tree  network  includes  garbage-collected  storage  as  a  standard 
part  of  the  programming  environment  it  supports  Garbage  collection  on  such  a  net¬ 
work  entails  the  identification  and  disposal  of  objects  that  will  never  be  used 
again  When  an  object  becomes  Inaccessible,  it  may  have  become  known  on 
several  processors  None  of  these  processors  can  take  the  initiative  to  delete  the 
object  outright  because,  in  general,  none  knows  whether  accessible  references  to 
the  object  exist  on  other  processors  Therefore,  it  seems  that  it  might  be  very 
difficult  to  ever  reclaim  the  object  If  the  object  is  only  known  on  one  processor. 
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the  story  Is  different  In  this  case.  It  is  obvious  that  no  references  to  the  object 
exist  on  other  processors  (else  the  reference  tree  would  be  larger)  and  therefore 
the  object  can  be  deleted  it  it  is  not  accessible  on  the  one  processor  where  It  is 
known 

Our  garbage-collection  scheme  works  by  shrinking  the  reference  tree  of  on 
object  to  be  collected  until  only  one  processor  knows  about  it,  at  which  point  the 
object  can  be  collected  by  traditional  means  In  order  tor  a  reference  tree  to 
shrink,  nodes  must  remove  themselves  from  it  Clearly,  any  node  which  has  more 
than  one  neighbor  in  the  reference  tree  cannot  unilaterally  remove  itself  if  it  did, 
the  tree  would  become  partitioned,  since  those  nodes  which  were  originally  con¬ 
nected  by  the  removod  node  would  now  have  no  moans  of  communicating  thus 
only  "leaf"  nodes  nodes  which  have  exactly  one  neighbor  also  in  the  reference 
tree  may  disconnect  themselves  from  it"  fortunately,  since  reference  trees  are 
acyclic,  every  reference  tree  has  leaf  nodes 

this  garbage-collection  scheme  depends  on  the  tact  that  a  garbage-collectable 
object  will  not  be  used  anywhere  once  it  becomes  garbage-collectable  Thus,  after 
some  Interval,  processors  with  references  to  such  an  object  may  guess  that  it  can 
be  collected  by  the  fact  that  they  have  not  seen  it  used  recently,  (ven  If  an 
object  is  still  potentially  accessible,  it  is  inefficient  to  keep  it  on  processors  where 
it  is  not  needed  Therefore,  it  is  economical  for  a  processor  in  the  reference  tree 
for  such  an  object  to  remove  itself  if  it  can.  If  that  strategy  is  applied  con¬ 
sistently.  the  reference  tree  of  any  garbage-collectable  object  should  slowly  shrink 


*An  additional  consideration  Is  that  no  leaf  node  which  Is  a  custodian  of  a  text 
for  an  object  may  remove  itself  from  that  object's  reference  tree  without  first 
preserving  the  text  by  passing  it  on  to  Its  neighbor 
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to  a  point  (a  single  node),  whereupon  the  object  can  be  disposed  of 

There  is  one  unfortunate  problem  with  this  scheme,  involving  the  collection  of 
objects  which  are  part  of  certain  cyclic  data  structures  for  example,  consider  an 
object  A  whose  text  contains  a  reference  to  B,  whose  text  in  turn  contains  a 
reference  to  A  Assume  further  that  the  structure  Is  garbage-collectable  that 
neither  A  nor  B  can  be  reached  from  any  active  event  Then  by  the  argument 
given  above,  the  reference  trees  for  both  A  and  B  should  slowly  shrink  to  a  point 
If  both  converge  to  the  same  point,  there  Is  no  problem  ordinary  garbage- 
collection  techniques  can  easily  handle  the  situation  However,  another  scenario  is 
possible,  as  outlined  in  figure  f>  /  Here  the  two  objects  may  spend  forever  chas¬ 
ing  each  others'  tails,  and  it  may  bo  that  noither  reference  tree  will  ever  shrink  to 
a  point  The  reason  this  can  happen  is  that  when  a  text  is  moved  from  one  proc¬ 
essor  to  another  It  draws  with  it  the  reference  trees  for  all  objects  referenced  in 
that  text  Thus  when  the  reference  tree  for  object  A  shrinks  and  the  text  of  A 
moves  from  processor  1  to  processor  2.  the  reference  tree  for  B  will  be  extended 
by  the  addition  of  a  link  from  processor  1  to  processor  2  If  the  reference  tree  for 
B  attempts  to  contract  next  by  the  removal  of  processor  3,  the  text  of  B  will  have 
to  be  sent  from  3  to  1 .  re  extending  the  reference  tree  of  A  to  include  processor 
1  Of  course,  there  are  many  other  sequences  of  events,  even  starting  from  one 

of  the  configurations  shown  in  Figure  fi  f,  which  will  result  in  both  reference  trees 

converging  on  the  same  point,  however,  it  is  possible  to  have  an  infinitely  long 
sequence  of  events  which  never  results  in  either  object  being  collected 

This  problem  is  similar  to  the  problem  of  cyclic  restart  in  some  transaction- 

based  data  base  management  systemsf3?],  perhaps  there  are  solutions  to  this 

garbage-collection  problem  which  are  analogous  to  solutions  to  the  cyclic  restart 
problem  There  is  reason  to  believe,  however,  that  the  problem  will  rarely  occur  in 
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Solid  lines  do  nolo  links  in  th*'  referent  e  troo  of  object  A,  rfa.sbecf  linos 
the  tree  for  object  B  /I  solid  bo *  represents  a  processor  with  a  tent  for  A, 
a  shaded  bon  a  processor  with  a  lent  lor  B  Successive  reference  tree  con¬ 
tractions,  alternating  between  the  reference  trees  of  A  and  B,  can  lead  to  the 
sequence  of  situations  shown  above  as  (a)  through  ( 1 ).  whereupon  a  final 
contraction  involving  B  will  restore  situation  (a). 

Figure  6  /  Cyclic  restart  in  garbage  collection 


actual  operation 


5.3:  The  State  Protocol 

Reference  trees  are  maintained  by  means  of  an  interprocessor  communication 
protocol  which  may  bo  used  to  grow  and  shrink  reference  trees  while  preserving 
the  required  connectedness  and  freedom  from  cycles  In  order  to  participate  in 
this  protocol,  each  processor  maintains,  for  each  object  it  knows  about,  a  link  state 
for  each  neighbor  (there  are  thirteen  different  link  states)  Various  flavors  of  mes¬ 
sages  can  be  sent  pertaining  to  an  object's  reference  tree,  and  a  simple  state 
transition  table  dictates  the  responses  so  as  to  keep  the  various  link  states  mutu- 
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ally  consistent  By  sending  appropriate  messages,  it  is  possible  tor  a  node  to 
delete  itself  from  a  reference  tree  (an  option  only  available  to  leaf  nodes)  or 
extend  a  reference  tree  to  include  a  new  processor  typically,  a  node  will  attempt 
to  delete  itself  If  it  discovers,  in  the  course  of  a  garbage  collection,  that  it  no 
longer  has  any  accessible  references  to  an  object  The  reference  tree  for  an 
object  will  be  extended  it  a  text  referencing  the  object  is  sent  to  a  processor  not 
previously  a  member  of  the  object’s  referenc  e  tree 

The  protocol  for  accomplishing  these  things  we  shall  call  the  mt^mtwr  shi  p  pro¬ 
tocol  (because  it  keeps  track  of  membership  in  reference  trees)  to  distinguish  it 
from  other  protocols  such  as  that  used  for  moving  object  texts  The  membership 
protocol,  a  f  irther  evolution  of  an  earlier  protocol[  T  5],  Is  more  intricate  than  one 
might  at  first  imagine  necessary,  but  simpler  protocols  failed  because  of  deadlocks 
or  inconsistent  states  reached  after  inopportune  sequences  of  events  The  proto¬ 
col  is  described  below  in  what  some  may  find  to  be  daunting  detail,  the  reader 
who  finds  the  goinq  tedious  can  at  any  point  skip  the  remainder  of  the  section  and 
proceed  to  the  next  without  any  loss  of  continuity  The  reader  who  perseveres, 
however,  will  hopefully  be  rewarded  with  more  than  merely  an  understanding  of  our 
particular  reference  tree  management  protocol  There  is  not  really  a  reference 
tree  protocol,  rather  there  is  a  whole  family  of  such  protocols,  a  representative 
member  of  which  is  dnsi  ribod  here  The  description  below  includes  an  outline  of 
the  destgn  considerations  which  made  the  current  protocol  what  it  is  After  com- 
pleVng  this  section,  the  reader  should  be  equipped  to  design  his  own  reference 
treo  protocols  to  fit  his  own  particular  needs,  or  even,  perhaps,  improve  on  the  one 
presented  here  ’he  important  thinq  about  this  description  is  not  its  detail,  but  its 
illustration  of  the  concept  of  using  link  states  to  maintain  a  global  structure  while 
keeping  only  local  information  The  material  in  this  section  is  presented  in  a 
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relatively  informal  stylo,  Appendix  H  contains  a  more  rigorous  argument  for  the 
correctness  of  the  membership  protocol  that  it  indeed  keeps  reference  trees 
connected  and  prevents  cycles  from  forming  in  them 

I  he  membership  protocol  makes  no  attempt  to  recover  from  damaged  or  lost 
messages,  or  messages  arriving  out  of  order  These  problems  can  he  solved  by 
various  well  known  mear.s[  .’6 )  which  may  be  assumed  to  provide  an  underlying  pro¬ 
tocol  on  each  link 

Ihe  membership  protocol  requires  that  each  object  have  a  globally  unique 
naan-  Ibis  name  s  needed  so  that  when  an  attempt  is  made  to  extend  a  refer¬ 
ent  e  tree  t  a  new,  pror  essor,  that  processor  can  determine  whether  it  already 
knows  of  the  object  via  some  other  route  This  information  in  turn  is  necessary  to 
detect  attempts  to  form  cycles  in  the  reference  tree 

The  membership  protocol  involves  seven  basic,  kinds  of  messages,  whose  mean 
ing  and  format  are  as  given  in  T  ahlo  5  8  1  ach  message  is  specialised  hy 
identification  of  the  object  (and  hence  reference  tree)  to  which  it  pertains  This 
specialisation  is  effected  in  one  of  two  ways,  depending  on  the  message  type 
Messages  which  establish  new  communication  paths  for  an  object  (commonly  by 
extending  the  object’s  reference  tree  to  include  another  processor)  include  the 
object's  global  name  (shown  as  C,N  in  Table  5  H)  These  messages  also  include  a 
shorter  /o-  a/  name  (IN)  which  the  sender  of  the  message  will  use  in  the  future  tor 
sending  references  to  the  object  over  the  same  communication  link  The  other 
messages  in  the  membership  protocol  (as  well  as  nil  other  messages,  including,  for 
example,  text  management  messages)  use  only  the  local  name  to  denote  the 
intended  object  The  use  of  local  names  not  only  shortens  messages  but  speeds 
their  processing  Since  the  space  of  local  names  is  smaller  and  presumably  reason¬ 
ably  compact,  conversion  of  an  incoming  local  name  to  an  internal  object  reference 
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can  bo  accomplished  economically  by  means  of  a  direct  table  look-up,  rather  than 


the  more  expensive  scheme  required  to  look  up  a  global  name 


Message 

Meaning 

R*  GN  t  N 

request  to  add  link  to  tree 

l*  GN  IN 

agreement  to  add  link 

l  GN  IN 

reftisal  to  add  link 

♦  IN 

transfer  mastery  of  link 

-  IN 

request  to  drop  link  from  tree 

A*  IN 

positive  acknowledgment 

A  IN 

nogativo  acknowledgment 

Table  ft  8  Membership  protocol  message  types 

Messages  In  the  membership  protocol  may  be  sent  either  spontaneously  (r.e.,  In 
response  to  some  external  stimulus,  such  as  the  need  to  hove  a  local  name  for  an 
objoct  so  that  a  text  referencing  it  can  be  sent)  or  In  response  to  an  Incoming 
message  requesting  some  change  In  a  reference  tree  R*.  ♦.  and  messages  are 
always  sent  spontaneously,  I  ♦.  I  ,  A*.  and  A  messages  are  always  sent  os 
responses 

The  membership  protocol  operates  by  associating  with  each  end  of  each  link  a 
state  for  each  possible  object  It  is  these  states  which  actually  define  the  extent 
of  an  object's  reference  tree  In  terms  of  implementation,  each  processor  must 
maintain  a  data  base  for  each  object  it  has  a  reference  to,  indicating  the  state 
(with  respect  to  that  object)  of  each  link  adjacent  to  the  processor  It  Is  impor¬ 
tant  to  realize  that  the  two  processors  at  the  ends  of  a  link  may  have  different 
Ideas  of  the  state  of  the  link,  this  may  be  as  the  result  of  some  Intentionally  intro¬ 
duced  asymmetries  discussed  below,  or  It  may  occur  if  messages  regarding  the 
objoct  have  been  sent  at  one  end  of  the  link  but  not  yet  received  at  the  other 

The  possible  states  may  be  grossly  characterized  as  being  either  stable  or 
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transient  Stable  states  are  states  which  might  be  expected  to  persist  over  a 


relatively  long  period  of  time  Transient  states  are  those  in  which  a  message  has 
been  sent  across  the  link  and  a  reply  is  expected;  the  reply  will  cause  a  transi¬ 
tion  to  some  other  state,  either  stable  or  transient  Transient  states  exist  to  pro¬ 
vide  the  proper  sequencing  so  that  the  next  pair  of  stable  states  to  be  esta¬ 
blished  is  consistent  and  does  not  result  in  partitioning  the  tree  or  closing  a  cycle 
for  purposes  of  discussion,  the  states  have  been  given  one-  to  three-choracter 
mnemonic  names  (listed  in  Table  f>  9.  below)  Tor  purposes  of  actually  manipulating 
object  references  passing  over  links,  the  most  important  attribute  of  each  link 
state  is  whether  a  processor  in  that  state  is  directly  prepared  to  send  or  receive 
a  local  name  for  the  object  on  that  link  Processors  in  all  states  but  X,  N.  and  N?1 
are  directly  ahlt*  to  send  local  names  (/.«?..  such  names  have  already  been  declared 
by  R*.  t  ♦.  or  1  messages);  processors  in  all  states  but  X,  N  and  are  directly 
able  to  receive  them  (;.e  ,  such  names  have  already  been  declared  to  them  and 
recorded) 

In  addition  to  the  various  link  states,  each  processor  maintains  for  each  object 
a  processor  state,  either  "in  the  reference  tree"  or  "not  in  the  reference  tree  " 
Certain  link  states  are  only  consistent  with  a  particular  processor  state  Rather 
than  show  the  pair  (processor  state,  link  state)  that  governs  a  processor's 
response  to  messages  arriving  on  a  link,  we  encode  the  processor  state  information 
into  the  link  state,  adopting  the  convention  of  using  state  names  containing  the 
letter  "X"  to  imply  a  processor  state  of  "not  in  the  reference  tree"  and  other 
state  names  to  imply  the  opposite  Thus  either  all  of  a  processor's  link  states  for 
a  given  object  will  contain  X's,  or  none  will  When  a  link  state  changes  between 
these  categories,  other  link  states  in  the  processor  must  also  change  to  preserve 
this  consistency  Special  transitions  (between  X  and  N,  X?  and  N?),  requiring 
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neither  the  receipt  nor  the  sending  of  messages,  are  provided  to  fulfill  this  need 


In  general,  each  X  link  state  has  an  analogous  N  state,  differing  only  in  the  proces¬ 
sor  state  of  the  processor  m  question 

We  now  describe  the  five  stable  states  Perhaps  the  state  most  likely  to 
occur  is  X,  which  Indicates  that  not  only  is  the  link  not  considered  part  of  the 
reference  tree,  the  processor  Is  not  considered  part  of  the  reference  tree  If  a 
processor  has  no  knowledge  of  an  object,  It  acts  as  though  it  were  in  state  X  for 
that  object  on  every  link 

The  state  analogous  to  X  is  N  In  state  N,  the  processor  is  considered  part  of 
the  reference  tree,  but  still  does  not  believe  the  link  in  question  to  be  part  of  the 
object's  reference  tree  State  N  may  come  about  either  because  the  processor  Is 
the  only  processor  to  contain  any  references  to  the  object,  or  because  the  proces¬ 
sor  is  connected  to  the  reference  tree  by  some  other  link  or  links 

Another  closely  relatod  state  is  L  State  L  is  like  state  H  in  that  the  relevant 
link  is  not  considered  |>art  of  the  reference  tree,  but  indicates  that  the  processor 
at  the  other  end  of  the  link  does  know  about  the  object,  and  that  local  names  have 
been  established  for  communicating  references  to  the  object  over  that  link  In  a 
stable  condition,  the  processor  at  the  other  end  of  the  link  will  also  be  in  state  L 
with  respect  to  the  link  The  reason  for  state  l  is  to  enable  the  communication  of 
an  object  reference  over  links  that  cannot  be  allowed  to  join  the  object's  refer¬ 
ence  tree  because  adding  those  links  would  close  cycles  Such  communication  is 
not  only  desirable,  but  sometimes  is  necessary. 

Another  stable  state  is  M,  which  indicates  that  the  link  In  question  is  believed 
to  be  part  of  the  reference  tree,  and  furthermore  that  this  processor  is  currently 
the  master  of  that  link  (for  transactions  involving  that  object).  The  master  of  a  link 
is  the  only  onn  that  can  effect  changes  in  the  status  of  the  link  or  send  a  text  of 
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the  object  over  the  link  1  his  asymmetry  seems  to  be  necessary  to  prevent  contu¬ 
sion  resulting  from,  for  Instance,  both  ends  of  a  link  simultaneously  attempting  to 
terminate  their  connection  with  the  reference  tree. 

In  a  stable  condition,  the  state  at  the  other  end  of  a  link  from  M  will  be  S,  tor 
"slave  "  A  processor  in  state  S  connot  directly  cause  a  change  In  the  status  of 
the  link;  it  may  however  (by  means  of  a  message  not  discussed  here)  request  the 
master  to  commence  a  change,  and  it  may  respond  to  messages  sent  by  the  mas¬ 
ter 

Ihe  transient  states  will  not  be  described  to  the  same  level  of  detail  as  the 
stable  states  lor  the  most  part,  they  acquire  their  meaning  from  their  relation¬ 
ships  with  the  stable  states  Instead  of  attempting  to  describe  the  meaning  of 
these  states,  we  present  a  state-transition  table  (Table  5  9),  and  summarize  below 
the  normal  sequences  for  handling  several  situations 


transition  Upon  Receiving 

Spontamrous 

State 

«♦  l  ♦  l  ♦  A*  A  IH 

t  ransitions 

X 

1  ♦  S 

N 

N 

1  1  L 

R*:M?  X 

L 

A  N?  1  A  ♦  M  L 

L? 

M 

M 

X’  ♦  S 

S 

M  a  kr7 1  .9 

X? 

A  X  NT 

N? 

N! 

a«  mr  ni 

N?  , 

an  n? 

X? 

N^l 

1*  S  N  N?  1 

R*  M-7  1 

l? 

_ A-N*M  A*.*  AN  :C7 

M*7 

1!  In  hl 

mm 

A*  S'7  W7  mm 

s *» 

S-7  s  s? 

I  he  notation  a.b  means  that  under  the  spe<  ifunl  circumstances,  a  transition  to  state 
b  can  occur  with  the  emission  of  message  a.  transitions  occasioned  by  the  rer'.eipt 
of  a  local  name  are  shown  in  column  I  N. 

Table  6  9  Membership  protocol  state  transition  table 
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Although  the  table  indicates  which  states  may  receive  a  local  name  and  what  state 
changes  may  ensue  from  that  eventuality  (shown  in  the  column  headed  IN),  it  does 
not  show  in  what  states  a  local  name  may  spontaneously  be  sent  As  mentioned 
above,  a  local  name  may  be  sent  from  any  state  except  X.  X?,  N,  and  N?1,  and 
never  causes  a  state  change  in  the  sender  If  a  processor  in  state  N  or  N?1 
wishes  to  send  a  reference  to  the  object,  It  must  first  send  an  R«  message,  which 
will  cause  a  state  change  to  a  state  from  which  a  local  name  may  be  sent  A  proc¬ 
essor  in  state  X  or  X?  has  no  business  sending  n  reference  to  the  object,  since 
such  a  processor  is  not  in  the  reference  tree  for  the  object  and  therefore,  presum¬ 
ably,  has  no  references  to  send 

Ihe  fundamental  principle  that  motivates  this  protocol  design  (other  than  the 
need  to  maintain  the  link  data  base  in  a  consistent  state)  is  that  a  processor  must 
always  be  able  to  send  a  reference  over  any  link  without  proarr&ngerrH'nt  for 
example,  it  is  not  acceptable  that  the  sending  of  a  reference  should  be  the  culmi¬ 
nation  of  some  transaction  allowing  the  reference  to  be  sent  only  upon  receipt  of 
suitable  clearance  from  the  message's  target  In  order  to  understand  this  require¬ 
ment,  the  circumstances  under  which  references  may  be  sent  must  be  considered 

In  general,  a  reference  will  be  sent  as  part  of  some  text  which  is  being  com¬ 
municated  between  processors.  Sending  a  text  involves  communicating  the  refer¬ 
ence  to  the  object  whose  text  is  being  sent,  as  well  as  references  to  other 
objects  referred  to  in  the  text.  Thus  obtaining  clearance  to  send  a  text  may 
involve  simultaneously  obtaining  clearance  to  send  several  object  references 
Unless  every  processor  always  has  clearance  to  send  any  object  reference,  It  is 
easy  to  see  how  the  piecemeal  aggregation  of  such  clearance  could  lead  to  a 
deadlock  on  a  link  This  is  especially  true  when,  as  is  the  case  with  this  protocol, 
transactions  involving  different  objects  are  completely  independent  no  overall 
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master-slave  relationship  applies  to  all  communication  on  a  particular  link,  tor  exam¬ 


ple 

Ihis  need  to  avoid  deadlock  is  one  of  the  primary  factors  acting  to  complicate 
the  protocol  design,  and  requires  that  any  processor  always  he  able  to  send  any 
object  reference  without  the  possibility  of  confusing  the  processor  at  the  other 
side  The  only  exception  to  this  requirement  occurs  if  the  sending  processor  is  in 
state  X  if  a  processor  has  no  references  to  an  object,  it  has  none  to  send1  In 
any  other  state,  the  processor  must  either  already  have  a  local  name  for  the 
object  to  use  over  the  link,  or  have  the  option  of  picking  a  local  name,  declaring  it 
to  its  neighbor  with  an  11+  message,  and  then  immediately  using  it  in  messages 

We  now  turn  to  how  and  why  various  state  changes  may  occur  We  start  with 
a  processor  in  state  X,  having  no  references  to  the  object  in  question  The  only 
kind  of  message  that  can  be  received  in  state  X  is  an  R*  message  from  some  proc¬ 
essor  attempting  to  extend  the  reference  tree  tor  the  object,  perhaps  in  order  to 
send  a  text  mentioning  the  object  Upon  receipt  of  the  R*  message,  our  processor 
returns  an  l  ♦  message  as  an  Indication  that  the  link  should  indeed  be  added  to  the 
tree,  and  changes  to  state  S  in  anticipation  of  the  sender  of  the  R  +  message 
entering  the  M  (master)  state  when  it  receives  the  l*  message  Simultaneously, 
the  states  of  all  other  links  to  our  processor  change  from  X  to  N,  indicating  that  our 
processor  Is  now  part  of  the  reference  tree  Also,  any  links  m  state  X7  change  to 
N7 

Now  that  our  processor  is  part  of  the  reference  tree,  it  may  attempt  to  further 
extend  the  tree  by  sending  a  reference  along  one  of  the  links  just  converted  to 
state  M  From  state  N  an  object  reference  must  be  preceded  by  an  R*  message. 
Upon  sending  the  R*  message,  the  sender's  state  for  that  link  changes  from  N  to 
the  transient  state  M7,  awaitirg  a  reply.  The  reply  to  R*  depends  on  the  condition 
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of  the  processor  at  the  other  end  of  the  link  If  it  was  in  state  X,  it  changes  to  S 
and  replies  with  L*,  as  described  above  Upon  receiving  the  L+  message,  our  proc¬ 
essor  changes  from  M?  to  M,  and  the  link  has  been  established  It  the  other  proc¬ 
essor  is  in  state  N,  then  the  link  cannot  be  added  to  the  reference  tree  because  it 
would  close  a  cycle  (since  both  processors  are  already  connected  by  some  other 
route  in  the  reference  tree)  Consequently,  the  other  processor  responds  nega¬ 
tively,  with  an  l  message,  and  changes  to  state  L  When  the  sender  of  the  H  + 
message  receives  the  l  message,  It  also  enters  state  L  As  long  as  both  proces¬ 
sors  remain  in  state  L,  local  names  have  been  established  for  communicating  refer¬ 
ences  to  the  object  over  the  link,  even  though  the  link  has  been  agreed  not  to  be 
in  the  object's  reference  tree. 

Another  possible  scenario  is  that  two  processors,  both  in  state  N  (for  ttie  same 
link)  might  simultaneously  attempt  to  add  that  link  to  the  tree  by  sending  R*  mes¬ 
sages  to  each  other  and  entering  state  M7  Under  these  circumstances,  it  is  clear 
that  the  link  should  not  be  added,  or  a  cycle  will  be  formed  Therefore,  each  M? 
will  react  to  the  R  +  with  a  transition  to  state  L 

Once  it  has  linen  agreed  that  a  link  is  part  of  the  reference  tree  for  an  object 
and  things  have  settled  to  a  quiescent  state  (r.e..  no  messages  are  in  transit),  one 
processor  (the  master)  will  be  in  state  M  and  the  other  (the  slave)  in  state  S  It 
is  a  simple  matter  to  reverse  the  roles  of  master  and  slave,  but  the  transaction 
must  be  initiated  by  the  master  The  master  sends  a  ♦  message  and  enters  state 
S  When  the  slave  receives  the  ♦  message,  it  enters  state  M 

Having  seen  how  a  link  may  be  established  in  the  reference  tree,  we  now 
come  to  the  question  of  how  a  link  may  be  deleted  from  the  tree  Due  to  the  con¬ 
nected.  acyclic  nature  to  the  tree,  every  time  a  link  is  deleted,  a  node  is  also 
being  removed  from  the  tree  Thus  the  only  reason  for  deleting  a  link  is  because  a 
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processor  wants  to  remove  itself  from  the  reference  tree  This  In  turn  will  be 
caused  by  that  processor's  discovery  that  it  has  no  references  to  the  object 
reachable  from  any  active  data  on  that  processor  No  node  which  has  more  than 
one  neighbor  in  a  reference  tree  can  unilaterally  remove  itself  If  it  did,  the  tree 
would  become  partitioned,  since  those  nodes  which  were  originally  connected  by 
the  removed  node  would  now  have  no  means  of  communicating  Only  "leaf" 
nodes  nodes  which  have  exactly  one  neighbor  also  in  the  reference  tree  may 
disconnect  themselves  from  it  furthermore,  no  node  may  remove  itself  from  a  tree 
leaving  dangling  (though  non-tree)  links  in  state  L  (but  a  link  inconveniently  in 
state  L  may  he  removed  by  sending  a  message  and  changing  to  state  L?  the 
reader  can  follow  the  transitions  that  ensue)  Ihus  a  processor  may  attempt  to 
remove  itself  from  the  tree  only  if  all  its  links  hut  one  are  in  state  N  (or  N?)  Addi¬ 
tionally,  that  one  link  must  be  in  state  M.  if  the  processor  is  currently  a  slave  on 
that  link  (for  that  object),  it  must  first  induce  the  master  of  the  link  to  relinquish  its 
mastery 

A  master  requests  to  remove  itself  from  the  tree  by  sending  a  message  to 
its  slave  and  changing  to  state  X7  (simultaneously  all  N  links  from  that  processor 
should  change  to  X  and  all  N?  links  to  X?)  The  slave  responds  with  A  and 
changes  to  state  N?1  Upon  receiving  the  A  ,  the  old  master  returns  to  state  X, 
emitting  another  A  When  it  receives  this  A  ,  the  old  slave  goes  to  state  N  from 
W'M  The  extra  level  of  acknowledgment  here  is  needed  because  a  processor  in 
state  S  may  send  out  object  references  as  local  names,  a  capability  it  must  have 
The  old  master  must  be  prevented  from  returning  to  state  X.  where  such  refer¬ 
ences  will  not  he  accepted,  until  it  is  confirmed  that  the  old  slave  is  no  longer  in 
state  S  In  effect,  the  A  message  sent  by  the  old  slave  serves  to  "flush"  the 
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channel,  bringing  up  the  rear  for  any  local  names  that  might  have  been  sent. 

A  complication  for  this  scheme  occurs  precisely  when  a  processor  In  state  S 
sends  such  a  local  name  to  an  ex-master  now  in  state  X?  In  this  case,  the  ex¬ 
master  will  once  again  be  in  possession  of  a  reference  to  the  object,  and  must 
abort  its  Initiative  to  leave  the  reference  tree  It  does  this  by  changing  to  state 
N!  upon  receiving  the  local  name  In  state  Nl,  when  the  expected  A  acknowledg¬ 
ment  is  received  from  the  old  slave,  the  reply  will  instead  be  an  A*  message  and  a 
transition  to  M7,  indicating  a  desire  to  remain  in  the  reference  tree  after  all  When 
the  old  slave,  now  in  state  N?1,  receives  the  A*  message,  It  replies  with  t+  and 
returns  to  state  S  Receipt  of  the  l  ■«  message  by  the  old  master  will  then  cause  it 
to  return  to  state  M  An  L ♦  message  Is  used  here  rather  than,  say,  A+,  because  a 
processor  in  state  N71  does  not  have  a  local  name  it  can  use  immediately  to  send 
references  over  the  link  The  I  ♦  message  serves  to  re-establish  such  a  local 
name 

Other  transitions  in  Table  5.9  exist  to  take  care  of  other  pathological 
occurrences.  For  example,  the  old  slave,  while  waiting  in  state  N^l  for  either  an 
A*  or  A  reply,  may  discover  that  it  needs  to  send  out  a  reference  to  the  object 
Since  in  state  N71  it  has  no  local  name  for  so  doing,  it  must  declare  one  by  send¬ 
ing  an  R*  message,  which  is  accompanied  by  a  transition  to  The  reader  can 

follow  the  sequence  of  transitions  and  replies  triggered  by  this  R*  message,  and 
see  some  more  of  the  entries  in  Table  5  9  come  into  play 

Cases  like  this  are  another  source  of  complication  in  the  membership  protocol. 
Generally  speaking,  whenever  a  processor  can  undergo  a  spontaneous  transition  to 
another  state,  the  new  state  must  be  able  to  respond  meaningfully  to  any  message 
the  old  state  might  have  been  expecting  When  both  processors  at  the  ends  of  a 
link  are  susceptible  to  spontaneous  transitions,  adding  one  now  function  to  the 
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protocol  may  require  the  addition  of  several  new  transitions. 


S.4:  Modifications  to  the  Reference  Tree  Concept 

This  section  is  devoted  to  a  discussion  of  several  potential  problems  with  the 
reference  tree  concept,  along  with  some  possible  solutions  to  them. 

5.4.1:  Reference  Tree  Management 

Use  of  reference  trees  can  lead  to  two  kinds  of  inefficiencies.  Both  are  sug¬ 
gested  by  the  reference  tree  depicted  in  I  igure  b  10 


The  solid  box  represents  a  processor  with  a  copy  ol  the  object's  text; 
the  shaded  txix  represents  a  processor  inquiring  for  the  text. 

figure  6  10  A  non-optimal  reference  tree 

One  liability  is  that  reference  trees  may  not  always  grow  in  the  most  desirable 
shapes  In  figure  6  10,  the  shadr  d  processor's  inquiry  and  the  solid  processor's 
response  will  both  have  to  travel  the  entire  If  ngth  of  the  reference  tree,  when  in 
fact  the  processors  are  adjacent  Tho  other  liability,  which  can  also  occur  in  more 
"strotched-out"  reference  trees,  is  the  overhead  involved  in  keeping  reference 
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trees  connected  t  veil  If  the  two  end  processors  In  Figure  6.10  are  the  only  two 
which  continue  to  have  any  Interest  in  the  object,  all  the  intermediate  processors 
must  still  stay  in  the  object's  reference  tree  to  keep  It  connected  In  a  large  sys¬ 
tem  with  many  objects,  such  extended  reference  trees  could  Impose  significant 
overhead  on  each  processor  Jhe  amount  of  overhead  involved  in  membership  in  a 
reference  tree  is  not  large,  but  is  non-zero  As  a  network  grew,  it  would  be  possi¬ 
ble,  especially  given  on  unfortunate  event  and  object  distribution  strategy,  for  the 
average  size  of  reference  trees  to  increase  to  the  point  where  each  processor 
found  itself  forced  to  keep  track  of  more  and  more  objects  Conceivably  the  net¬ 
work  could  be  slowly  strangled  as  more  and  more  of  its  storage  was  devoted  to 
these  reference  tree  "cobwebs  "  Lven  more  disturbing  is  the  prospect  that  a  simi¬ 
lar  fate  would  befall  a  network,  not  because  of  any  attempt  to  scale  up  the  number 
of  processors,  but  simply  because  of  the  accretion  of  objects  over  time,  as  users 
come  to  have  more  and  more  data  stored  on  the  system 

5. 4. 1.1:  Disconnecting  Reference  Trees 

Overhead  imposed  on  intermediate  processors  in  a  reference  tree  by  the 
requirement  that  the  tree  be  kept  connected  might  be  eliminated  if  it  were  permis¬ 
sible  to  disconnect  pieces  of  a  reference  tree  It  is  not  difficult  to  devise  a  proto¬ 
col  by  which  one  of  the  Intermediate  processors  could  cause  this  to  happen  Since 
all  communication  involving  an  object  travels  only  along  links  in  its  reference  tree, 
however,  two  requirements  must  be  mot  a  custodian  of  a  copy  of  the  text  of  the 
object  must  exist  on  either  side  of  the  break  (otherwise  the  processors  in  one  of 
the  disconnected  pieces  would  have  no  access  to  the  text  of  the  object),  and  the 

i. 

object  must  be  immutable  (otherwise  an  update  performed  in  one  half  of  the  tree 
would  never  become  visible  in  the  other  half)  Generally  speaking,  it  is  not  possible 


13R 


Chapter  5:  Reference  Trees 


to  tell  whether  a  side  effect  may  in  the  future  be  performed  on  an  object,  limiting 
the  applicability  of  this  approach.  However,  It  such  information  were  supplied,  say 
In  the  form  of  a  "read-only"  bit,  reference  trees  of  “read-only"  objects  could  be 


disconnected 

Lftectively,  breaking  a  link  in  the  reference  tree  for  an  object  creates  two 
reference  trees  for  the  object  I  ach  of  the  new  trees  will  then  behave  as  if  it 
were  the  only  reference  tree  for  that  object  Specifically,  leaf  nodes  of  either 
tree  may  then  delete  themselves  from  it,  so  all  the  intermediate  processors  in  our 
example  can  leave  the  tree,  one  by  one,  resulting  in  the  desired  situation  where 
only  the  two  distant  processors  know  about  the  object 

In  fact,  the  only  problem  with  disconnecting  a  reference  tree  arises  if  it  is 
ever  desired  to  re-connect  the  tree  I  fie  reference  tree  management  protocol 
avoids  cycles  in  reference  trees  by  refusing  to  make  a  connection  if  two  branches 
of  reference  tree  for  the  same  object  bump  into  each  ottier  This  is  done  because 
it  is  assumed  ttiat  all  branches  of  reference  tree  for  the  same  object  are  already 
connected,  therefore,  adding  another  connection  would  close  a  cycle  If,  as  the 
result  of  a  disconnection,  the  two  branches  are  not  connected,  the  protocol  will 
still  refuse  to  connect  them  It  would  tie  quite  difficult  to  allow  such  branches  to 
be  re-connected  without  introducing  the  possibility  that  cycles  could  be  formed 
Thus  it  seems  ttiat  once  a  reference  tree  is  broken  into  two  or  more  pieces,  'host* 
pieces  must  continue  to  exist  independently  for  as  long  as  they  continue  to  exist 
This  is  not  necessarily  bad,  though  1  ach  piece  is  still  free  to  grow,  shrink,  and 
move  just  as  the  original  was,  and  thus  each  may  independently  be  reclaimed  by 
the  garbage  collection  mechanism  when  its  usefulness  is  ended 
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5. 4. 1.2:  Reorganizing  Reference  Trees 

figure  5.10  gives  an  example  ot  a  non-optlmal  reference  tree  which  could  be 
improved  by  being  rerouted  Any  approach  to  rerouting  based  on  purely  local 
knowledge  of  the  reference  tree  should  fit  easily  into  our  scheme,  provided  It 
preserves  the  essential  properties  of  reference  trees  (connectedness  and  freedom 
from  cycles) 

One  possibility  is  for  a  leaf  node  which  is  aware  that  one  of  its  neighbors  Is 
connected  to  the  tree  by  a  different  path  (perhaps  because  of  being  in  state  L 
with  respect  to  that  neighbor)  to  break  its  old  connection  and  connect  instead  to 
that  neighbor  Ihis  kind  of  operation  is  depleted  in  I  igure  5.1  1. 


Figure  6. 1  1  A  simple  reference  tree  reorganization 

It  is  difficult,  unfortunately,  for  a  non-leaf  node  to  make  this  kind  of  Jump,  because 
it  will  not  know  which  of  its  old  links  to  break  If  the  wrong  choice  Is  made,  not 
only  will  the  reference  tree  become  disconnected,  but  one  halt  of  It  will  contain  a 
cycle,  as  shown  in  figure  5  12 

In  addition  to  the  mechanics  of  reorganizing  the  tree,  there  is  of  course  a 
strategy  question  when  is  reorganization  wise?  Once  again,  in  simple  cases  the 
answer  can  be  fairly  obvious.  but  in  general  It  may  not  be  If  the  goal  for  the  proc¬ 
essor  changing  its  links  is  to  get  closer  to  a  copy  of  the  text  of  the  object,  then  it 
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I  ink  Added 


figure  h  1  ?  A  dangerous  reference  tree  reorganization 


is  obviously  a  good  idea  to  reorganize  if  the  processor  being  connected  to  has  a 
copy  of  the  text  It  it  does  not  have  a  copy  of  the  text,  then  it  will  either  have  to 
have  some  idea  how  tar  the  nearest  text  is  (a  piece  of  information  that  might 
become  obsolete  every  time  a  toxt  moved)  or  other  considerations  will  have  to  be 
Invoked 
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5.4.2:  Reliability 


As  tho  number  of  components  In  a  system  grows,  the  likelihood  that  all  these 
components  will  be  operational  at  the  same  time  decreases  Although  much  of  a 
reference  tree  network  can  continue  to  function  in  spite  of  some  component 
failures,  such  failures  may  nevertheless  affect  tho  network  more  seriously  than  is 
desirable  or  necessary  for  a  moderatesl/ed  network,  this  need  not  be  a  major 
concern,  after  all,  there  are  many  large  centralized  computer  installations  with 
essentially  no  resiliency  against  failures,  at  least  In  tho  central  processor,  and 
their  overall  reliability  record  remains  acceptable  If  a  reference  tree  netwoik  is 
viewed  ns  a  replacement  for  one  of  these  machines,  there  is  no  <j  priori  reason  to 
believe  that  a  reference  troe  network  with  the  same  number  of  components  should 
he  any  more  prone  to  failure 

One  attrai  tion  of  reference  tree  networks,  however,  is  the  possibility  of  scal¬ 
ing  them  up  to  very  large  sizes,  where  the  failure  of  one  or  more  components  might 
he  a  common  occurrence  furthermore,  if  reference  free  networks  have  the  poten¬ 
tial  for  enhanced  reliability  through  appropriate  design  changes,  it  is  a  shame  not  to 
take  advantage  of  this  possible  benefit,  even  in  networks  of  modest  size.  Finally, 
it  may  often  he  desirable  to  plan  "failures’*  of  various  components  to  take  them 
temporarily  out  of  circulation  for  preventive  maintenance  or  reconfiguration.  It  turns 
out  that  even  such  planned  shutdowns  are  difficult  to  manage  with  the  basic  refer¬ 
ence  tree  scheme 

If  is  useful  to  categorize  failures  as  either  failures  of  nodes  or  failures  of  links 
connecting  nodes  The  basic  problem  that  arises  when  a  link  falls  is  that  all  refer¬ 
ence  trees  going  through  that  link  become  partitioned  operations  on  objects 
whose  reference  trees  do  not  include  that  link  will  not  he  affected  Tho  obvious 
solution  to  this  problem  is  to  devise  a  protocol  for  rerouting  the  affected  reference 
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trees  through  other,  still-functioning  links,  without  making  the  reference  tree  cyclic 


or  otherwise  leaving  the  reference  tree  information  in  an  inconsistent  state  A 
scheme  for  doing  this,  as  well  as  dealing  with  node  failures,  is  the  subject  of 
thesis  research  by  Clark  Haker[2] 

Node  failures  are  more  serious  than  link  failures  Not  only  does  a  node  failure 
look  like  a  failure  of  all  links  leading  to  that  node,  but  the  Integrity  of,  or  at  least 
access  to,  data  stored  at  that  node  may  tie  compromised  If  the  only  copy  of  an 
object's  text  is  stored  at  a  failed  node,  and  that  copy  Is  destroyed  in  a  failure,  It 
is  difficult  to  see  how  to  recover  it  If  that  problem  is  to  be  solved,  it  must  be 
solvod  either  by  redundant  storage  of  object  texts,  or  some  higher-level  mechanism 
for  regenerating  lost  objoct  texts,  or  both  Also  serious  is  the  problem  of  what  to 
do  with  events  that  may  have  been  stored  on  ttie  failed  processor,  and  how  to  tell, 
if  such  events  vanish,  whether  they  vanished  before  or  after  being  executed 
These  subjects,  too,  are  being  studied  by  Clark  (Taker,  but  it  remains  to  be  seen 
how  the  cost  of  avoiding  these  problems  will  trade  off  against  the  improvement  in 
reliability  obtained  thereby 

5.4.3:  Global  Names 

The  network  protocols  as  currently  envisioned  require  that  each  object  have  a 
unique  global  name,  assigned  when  the  object  is  created,  and  unchanged 
thereafter  This  global  name  is  used  in  determining  whether  references  that  arrive 
at  a  processor  by  different  routes  actually  refer  to  the  same  object  This  testing 
of  objects  for  identity  is  an  operation  which  the  user  himself  may  well  wish  to  per¬ 
form,  and  Is  also  necessary  if  cycles  in  reference  trees  are  to  bo  avoided 

Since  objects  are  created  asynchronously  at  several  nodes  in  the  network,  the 
problem  arises  of  making  sure  that  all  these  nodes  deal  out  unique  global  names. 
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(he  simplest  solution  is  to  partition  the  set  of  possible  global  names  among  the 
processors,  for  example  by  having  each  processor  Insert  a  unique  identifying  string 
Into  the  global  names  of  the  objects  it  creates  I  his  solution  sutlers  from  a  minor 
lack  of  expansibility,  since  the  space  of  unique  processor  ID's  must  be  large 
enough  to  accommodate  all  processors  which  might  be  added  to  the  system  in  the 
future  More  seriously,  after  some  period  of  time,  a  processor  will  run  through  all 
the  global  names  allotted  to  it  Lven  if  it  were  to  "borrow"  some  yet  to-beused 
names  from  other  processors,  after  a  long  enough  interval  all  global  names  will  have 
been  used  (he  problem  may  be  solved  for  practical  purposes  by  allowing  for  a 
name  space  large  enough  to  last  for  a  very  long  time,  but  it  can  also  be  solved  by 
"recycling"  the  global  names  of  objects  which  are  garbage-collected,  reusing  those 
names  for  new  objects  Then  the  global  name  space  need  only  be  as  large  as  the 
total  number  of  objects  that  can  exist  in  the  system  at  one  time  a  bound  that  Is 
difficult  to  improve  on  as  long  as  each  object  requires  a  global  name 

Aside  from  these  complications,  there  are  other  reasons  for  disliking  global 
names  for  example,  they  make  it  difficult  to  Interconnect  systems  that  previously 
had  been  operating  independently  It  Is  possible,  in  fact,  to  devise  a  reference 
tree  scheme  which  uses  no  global  names  Such  a  reference  tree  scheme  cannot 
easily  prevent  cycles  in  reference  trees,  but  the  kind  of  "cycles"  that  form  are  not 
fatal  They  resemble  helices  more  than  cycles,  with  successive  coils  of  a  helix 
falling  on  top  of  each  ottier  What  prevents  such  cycles  from  being  harmful  is  that 
each  time  »  coil  of  a  helix  passes  through  a  processor.  It  appears  as  a  different 
reference  Only  by  initiating  some  kind  of  "trace"  operation  can  a  processor  dis¬ 
cover  whether  two  such  references  actually  refer  to  the  same  object  This  possi¬ 
bility  that  an  object  might  manifest  itself  as  two  superficially  different  references, 
or,  put  another  way.  that  two  apparently  distinct  references  might  be  found  to 
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refer  to  the  same  object,  puts  a  new  wrinkle  into  the  semantics  of  VIM  *  Without 
this  mechanism,  it  can  be  determined  if  two  references  refer  to  the  same  object  by 
simply  checkiny  if  both  are  actually  the  same  reference,  a  relatively  inexpensive 
operation.  While  some  may  regard  this  checking  of  objects  for  identity  as  a  “dirty" 
operation,  it  is  often  a  useful  one  for  example,  the  implementation  of  a  LlSP-like 
language  will  be  a  good  deal  more  efficient  it  the  equivalent  of  the  LISP  LQ 
test[25]  can  bo  performed  by  simply  comparing  references  the  generation  of 
unique  objects  to  serve  as  keys  (as  in  Hendersonf  1  /’])  and  their  Inter  comparison  is 
another  example  of  a  situation  where  being  able  to  check  objects  for  identity  is 
useful 

In  conclusion,  although  our  scheme  for  doing  without  global  names  has  its 
attractions,  it  harbors  possible  inefficiencies,  as  the  reference  tree  for  just  one 
object  can  grow  without  bound  Only  experimentation  can  show  whether  this  kind 
of  helical  reference  tree  would  pose  a  serious  problem  in  practice 

5.4.4:  Non-Homogeneous  Networks 

So  far,  we  have  been  assuming  that  reference  tree  networks  are  homogene¬ 
ous  although  nodes  may  exhibit  various  differences  in  capacity  and  features,  all 
use  substantially  the  same  internal  representations  of  data  and  algorithms  This  is 
an  entirely  reasonable  approach  for  many  uses  of  reference  tree  networks,  espe¬ 
cially  with  increasing  standardization  of  computing  hardware  around  conventions 
such  as  8-bit  bytes  and  1  G-bit  words  However,  because  of  special  features  avail¬ 
able  or  advnncinq  technology,  it  may  become  desirable  to  include  various  non- 
conforming  processors  into  a  network.  Such  processors  could  probably  handle 

*The  "link"  mechanism  proposed  by  Gula[14]  adds  a  similar  wrinkle 
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differences  in  ilata  representation,  at  least  within  limits,  by  automatic  translations 
of  incoming  and  outgoing  object  texts  However,  it  might  be  difficult  to  translate 
texts  which  contain  executable  code  this  way  A  possible  solution  here  Is  to  main¬ 
tain  several  versions  of  such  texts,  so  that  a  processor  could  call  for  the  version 
of  a  text  which  suited  its  need  In  fact,  if  executable  texts  were  originally 
expressed  In  some  higher-level  language,  that  higher-level  version  might  be 
retained  as  well,  and  new  machine-level  versions  generated  upon  demand 

S.5:  Summary 

I  he  reference  tree  mechanisms  described  in  this  chapter  provide  a  complete 
set  of  capabilities  for  managing  objects  in  a  network  the  protocols  outlined  allow 
for  side  effects  to  be  performed  on  objects,  for  the  maintenance  of  multiple  copies 
of  objects,  and  for  garbage  collection  of  inaccessible  objects,  even  when  these 
have  become  known  across  several  processors 

A  primary  objective  of  the  reference  trrre  design  is  to  permit  an  extremely 
flexible  view  of  event  and  object  text  locations  By  keeping  down  the  overhead 
involved  in  moving  objects  and  events  around,  frequent  readjustment  of  their  loca¬ 
tions  is  encouraged  One  sense  in  which  this  overhead  is  low  is  that  all  updates  to 
the  reference  tree  data  base  are  strictly  local  and  asynchronous  Since  increases 
in  total  system  si/e  have  no  effect  on  the  overhead  of  any  particular  reference 
tree  transaction,  the  use  of  reference  t'ees  is  compatible  with  building  systems 
that  can  be  scaled  up  to  arbitrarily  large  sizes 

The  only  nonlocal  aspects  of  reference  tree  management  are  the  use  of  global 
names  for  objects  and  the  use  of  globally  unique  time  stamps  for  conflict  resolution 
The  cost  of  storing  these  names  and  time  stamps  increases  logarithmically  in  the 
si/e  of  the  network  This  increase  in  cost  is  the  only  crimp  in  the  unlimited 
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scalability  of  reference  tree  networks.  Section  b.4.3  suggests  a  way  of  doing 
without  global  names,  at  some  cost;  perhaps  an  analogous  scheme  exists  for 
resolving  conflicts  without  recourse  to  globally  unique  time  stamps 

Beyond  these  limits  on  the  scalability  of  reference  tree  networks,  the  refer¬ 
ence  tree  mechanisms  are,  at  least  in  theory,  prey  to  various  kinds  of  inefficiencies 
ami  unreliabilities  There  is  reasonable  hope,  however,  that  methods  proposed  in 
this  chapter,  and  others  like  them,  can  conquer  these  problems  well  enough  to 
make  reference  trees  truly  practical  on  a  large  scale 
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Up  to  this  point,  our  primary  concern  has  been  with  the  architectural  details  ot 
reference  tree  networks:  ensuring  that  necessary  facilities  are  present,  and  that 
they  will  operate  correctly  t specially  since  a  principal  motivation  of  reference 
tree  networks  is  economic,  tiowever,  we  cannot  be  satisfied  with  a  design  simply 
because  it  allows  the  expression  of  useful  algorithms  and  does  not  make  mistakes 
in  Interpreting  them  Accordingly,  this  chapter  is  devoted  to  understanding  perfor¬ 
mance  aspects  of  reference  tree  networks  We  begin  by  proposing  a  simple  model 
of  reference  tree  networks,  designed  to  highlight  variables  that  most  directly 
affoct  performance  The  meanings  of  various  measures  of  performance,  in  the  con¬ 
text  of  this  model,  are  then  discussed  finally,  conclusions  obtained  by  applying 
the  model  to  a  variety  of  hypothetical  situations  are  presented 

6.1:  Models  of  Network  Elements 

It  is  logical,  when  modeling  something,  to  mimic  its  gross  structure.  In  our 
model  of  reference  tree  network  hardware,  we  copy  the  overall  topology  of  the 
network,  substituting  idealized  models  for  processors  and  links  Our  task  is  thus 
reduced  to  that  of  constructing  suitable  idealizations  of  these  elements.  What  we 
would  like  to  do  is  characterize  the  resources  available  in  each  kind  of  network 
element  and  the  effect  of  each  network  activity  on  the  consumption  of  each 
resource 

For  a  processor,  the  resources  available  to  be  consumed  are  CPU  time  and 
memory  space  Memory  space  is  occupied  by  events  and  by  object  texts  (texts 
containing  both  data  and  algorithms).  CPU  time  is  needed  for  execution  of  events. 

CPU  time  is  also  used  for  the  encoding  of  events  and  object  texts  preparatory  to 
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stunting  them  across  a  link  to  another  processor,  as  well  as  tor  the  corresponding 
decoding  of  incoming  events  and  object  texts.  Other  sinks  of  CPU  time  exist  also, 
such  as  garbage  collection  and  interrupt  processing  load.  Since  the  relationship  of 
these  loads  to  the  primary  activities  of  the  processor  is  often  indirect  and  unclear, 
we  do  not  represent  these  loads  explicitly  In  the  model  to  the  extent  that  they 
are  constant,  these  loads  can  bo  accounted  for  by  reducing  the  total  CPU  time 
available  for  event  execution  and  message  processing,  to  the  extent  that  the 
loads  vary  with  the  frequency  of  any  activity  that  is  represented  in  the  model,  the 
time  taken  can  be  added  in  as  "overhead"  to  the  time  required  by  that  activity 
To  the  extent  that  other  factors  cause  these  loads  to  vary,  the  model  will  be  inac¬ 
curate 

The  model  for  links  is  influenced  by  an  expectation  that  most  reference  tree 
networks  will  exist  within  a  fairly  localized  area,  so  that  any  delays  associated 
with  message  traffic  will  be  due  primarily  to  bandwidth  limitations  and  not  to  physi¬ 
cal  delay  between  one  end  of  the  message  channel  and  the  other.  Accordingly, 
the  omy  resource  available  in  a  link  is  "message  time,"  which  is  used  up  by  mes¬ 
sages  (r.e.,  shipment  of  events  and  object  texts)  in  proportion  to  their  length  (We 
/ 

concentrate  exclusively  on  transmission  of  events  and  object  texts,  treating 
membership  protocol  messages,  inquiries,  and  the  like  as  overhead  for  links,  just  as 
garbage  collection  and  interrupt  processing  are  treated  as  overhead  for  CPU's 
This  simplification  is  generally  in  accord  with  the  relative  sizes  and  frequencies  of 
differe  t  kinds  of  messages  on  the  MuNet.)  However,  the  link  has  no  delay,  in  the 
sense  that  the  moment  a  message  begins  to  be  sent  at  one  end,  it  begins  to  be 
received  at  the  other,  and  the  moment  the  last  bit  has  finished  being  sent  at  one 
end,  it  has  finished  being  received  at  the  other.  Thus  a  link  resembles  an  orifice, 
or  narrow  bottleneck,  rather  more  than  a  pipeline.  This  is  a  fairly  accurate 
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mathematical  representation  of  communication  over  short  distances  using  a  parallel 
or  serial  line,  this  delayless  assumption  will  be  relevant  in  some  applications  of  the 
model,  but  for  others  the  results  would  not  be  affected  even  if  ttie  interprocessor 
messages  suffered  delay  as  well  as  a  bandwidth  limitation 

Our  first  excursion  into  model  building  will  ignore  memory  limitations  of  proces¬ 
sors,  along  with  space  constraints  in  general,  to  concentrate  solely  on  time  con¬ 
straints  Under  these  circumstances,  it  is  convenient  to  imagine  a  first-in- first-out 
(I  IPO)  message  buffer  of  unlimited  capacity  interposed  before  each  link  and  also 
after  each  link  Then  one  can  imagine  a  processor  able  to  compose  several  mes¬ 
sages  and  dump  them  into  the  f  IFO  preceding  the  link,  letting  the  messages  per¬ 
colate  at  their  own  pace  through  the  link  into  its  following  FIFO  I  ho  receiving 
processor  can  then  look  at  the  messages  as  they  come  in,  or  allow  them  to  accu¬ 
mulate  for  a  while  in  the  FIFO  *  Assuming  those  FIFO's,  which  may  be  thought  of  as 
message  buffers  within  the  processors  (but  accessed  directly  by  the  links), 
simplifies  application  of  the  model  to  situations  where  the  ability  of  a  processor  to 
keep  a  message  channel  full,  by  initiating  a  new  message  as  soon  as  the  channel 
becomes  free,  might  otherwise  be  in  question 

Our  model  of  links  is  unidirectional,  whereas  actual  reference  tree  links  are 
required  to  be  bidirectional  A  bidirectional  link,  of  course,  may  be  constructed  out 
of  two  unidirectional  links  running  in  opposite  directions,  but  our  unidirectional  model 
is  useful  in  examining  situations  where  we  wish  to  constrain  the  flow  of  events  and 


*Thls  Is  a  reasonably  accurate  representation  of  what  actually  goes  on  in  the 
MuNet 
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object  texts  over  a  link  to  be  in  only  one  direction  ^ 

The  final  model  is  perhaps  best  illustrated  by  figure  6.1,  which  shows  our 
model  of  a  network  composed  of  two  processors  and  one  link  connecting  them 

□ - □ 

(a)  A  simple  reference  tree  network.,  with  two  processors  and  one 
link. 
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(b)  A  model  of  the  network  shown  in  (a),  with  performance  parameters 
r,  s.  t.  r\  s',  t’,  m.  and  m\ 

figure  6.1:  The  reference  tree  network  model 

Inside  the  active  elements  of  the  model  are  shown  performance  parameters.  The 
parameter  m  (or  m')  of  a  link  may  be  thought  of  as  the  number  of  time  units  taken 
by  that  link  to  transmit  one  word  of  message  The  parameter  r  (or  r')  of  a  proces¬ 
sor  is  the  number  of  units  of  CPU  time  required  per  word  of  incoming  (or  received) 
message,  similarly,  s  (s')  gives  the  amount  of  CPU  time  used  per  word  of  message 
sent.  Finally,  f  (f‘)  indicates  the  speed  of  the  processor  for  event  execution  t 
may  be  thought  of  as  the  amount  of  CPU  time  consumed  per  unit  of  computation 
during  event  execution  The  primed  parameters  are  shown  to  emphasize  that  every 
element  of  the  network  may,  in  general,  have  different  sets  of  parameters,  indica¬ 
tive  of  ther  differing  capacities  to  perform  the  various  operations  that  may  be 

^Even  If  such  a  constraint  were  adopted  in  a  real  reference  tree  network,  how¬ 
ever,  the  reverse  channel  would  still  be  needed  for  reference  tree  protocol  mes¬ 
sages 
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required  of  them  The  parameters  r,  s,  f,  m,  etc.,  can  be  further  generalized  in 
terms  of  concepts  presented  in  the  next  section 

A  formal  description  of  a  network  topology  may  be  given  in  terms  of  a  set  N  of 

nodes,  a  set  L  of  links,  and  two  Boolean  matrices  /  and  O  An  element  /  is  1  it 

jm 

and  only  if  link  ;  t  l  enters  node  m  c  N.  else  is  0  Similarly,  an  element  Ojm  ls  1  '1 
and  only  if  link  j  leaves  node  m 

6.2:  A  Model  of  Computations 

Constructing  a  plausible  model  of  the  network  hardware  is  trivial  in  comparison 
to  the  task  of  modeling  the  computations  the  network  is  to  perform.  Many  computa¬ 
tions  of  real  practical  interest  have  anatomies  complex  enough  to  defy  any 
mathematical  modeling  approach  other  than  simulation  Among  the  computations  that 
can  be  usefully  modeled,  the  universe  of  alternatives  is  too  diverse  to  be  encom¬ 
passed  under  any  one  modeling  procedure  We  are  forced,  therefore,  to  commit 
ourselves,  at  least  temporarily,  to  studying  computations  belonging  to  some  fairly 
specific  class. 

the  class  of  computations  on  which  we  choose  to  concentrate  is  not  the  most 
interesting  class  of  possible  computations  for  a  reference  tree  network,  but  it  has 
the  virtue  of  being  fairly  simple  Furthermore,  it  can  be  extended  sufficiently  so 
that  it  is  at  least  plausible  to  argue  that  the  scheduling  strategy  which  performs 
the  best  on  computations  in  this  class  will  also  be  among  the  best  performers  in 
actual  practice,  even  when  applied  to  computations  outside  the  class.  In  other 
words,  significant  further  improvement  in  network  efficiency  may  only  be  achievable 
by  adopting  radically  different  approaches,  such  as  making  available  to  the  schedul¬ 
ing  algorithm  additional  precompiled  information  indicating  the  best  treatment  for 
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each  specific  computation 


This  class  of  computations  to  be  modeled  is  a  class  of  recurring  computations 
where  each  computation  has  fairly  simple  structure,  "Recurring"  is  inteodod  to  con¬ 
vey  the  notion  of  a  series  of  sets  of  Input  values  arriving  at  periodic  intervals  for 
processing  There  Is  little  or  no  interaction  between  the  processing  of  ore  set  of 
Input  values  and  the  processing  of  other  sets  of  values,  and  the  processing  of 
each  set  of  values  proceeds  along  a  similar  course.  An  example  of  such  a 
recurrent  computation  would  be  a  system  receiving  three-dimensional  vectors  and 
performing  a  perspective  transformation  and  windowing  to  yield  a  sequence  of  two- 
dimensional  vectors  One  could  imagine  a  steady  source  of  three-dimensional  vec¬ 
tors.  contributing  either  to  continued  drawing  of  a  picture  or  to  successive  frames 
of '  a  movie  In  either  case,  the  processing  undergone  by  each  vector  follows  the 
same  outline,  or  at  worst  has  a  few  variants  according  to  whether  or  not  the  vec¬ 
tor  turns  out  to  be  visible 

This  concentration  on  recurring  tasks  is  convenient  for  studying  the  ultimate 
performance  capabilities  >f  a  network  topology,  since  any  amount  cf  parallelism  that 
might  be  needed  in  order  to  use  the  ne’werk  resources  to  their  full  capacity  can 
be  obtained  by  simply  adding  more  concurrent  tasks  This  approach  carefully 
sidesteps  any  consideration  of  how  much  parallelism  can  be  used  in  the  solution  of 
an>  particular  task  In  many  other  situations  this  constraint  of  the  task  may  very 
well  be  a  much  more  significant  const'aint  on  performance  than  saturation  of  the 
network  elements  with  work  Nevertheless,  our  Interest  here  is  In  the  overheads 
and  limitations  involved  in  the  use  of  network  resources,  as  opposed  to  the  plausi¬ 
bility  of  actually  bringing  all  those  resources  into  play,  and  thus  we  proceed  with 
our  recurrent-computation  model 

Given  a  recurring  stream  of  input  data,  the  oue  .tion  becomes  how  to  model  the 
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computation  applied  to  each  element  of  the  stream  this  is  done  by  means  of 
events  and  transitions  An  event  represents  a  computation  in  a  certain  intermediate 
state,  and  corresponds  to  a  VIM  event  A  transition  represents  the  process  of 
swallowing  up  an  event  and  producing  zero  or  more  other  events  It  thus 
corresponds  to  an  event  execution  under  VIM  An  event  is  something  which  can  be 
passed  from  one  processor  to  another,  or  held  for  future  execution  The  important 
parameter  of  an  event  is  its  si/e,  which  determines  the  storage  required  to  hold 
the  event,  and  the  cost  of  shipping  it  between  processors  ’  A  transition  has  two 
basic  parameters  the  time  required  to  effect,  or  execute,  the  transition  and  the 
space  used  to  represent  the  algorithm  which  implements  the  transition  Of  these 
parameters,  execution  time  is  the  more  significant  for  the  purposes  of  this  chapter 

In  the  model,  events  fall  into  some  small  number  of  even?  classes  containing 
events  with  the  same,  or  at  least  statistically  similar,  properties  Similarly,  transi¬ 
tions  may  be  grouped  into  transition  classes  Two  of  the  event  classes  are  dis¬ 
tinguished  as  the  initial  event  class  and  the  final  event  class  Receipt  of  a  set  of 
values  to  be  processed  is  modeled  by  the  receipt  of  an  event  in  the  initial  event 
class,  containing  these  values,  at  some  processor;  output  of  a  set  of  result  values 
is  represented  by  the  sending  out  >t  a  corresponding  event  in  the  final  event 
class 

More  formally,  then,  our  model  recognizes  a  set  E  of  event  classes  f  ach 
event  class  has  an  associated  size  which  is  characteristic  of  events  in  that  class 
One  event  c'ass  ts  distinguished  as  the  initial  event  class  and  another  as  the  final 
event  cla"  Th  •  model  also  recognizes  a  s»t  T  of  transition  classes,  where  each 
transition  class  has  an  execution  time  and  algorithm  size  pertaining  to  transitions  in 

*lt  is  assumed  that  all  data  required  for  execution  of  an  event  Is  accounted 
for  in  calculating  the  "size"  of  the  event. 
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that  class  there  is  additionally  an  ExT  matrix  G,  such  that  G ^  gives  the  number 
of  events  of  class  t  consumed  (if  0^  <  0)  or  generated  by  a  transition  of  class  A 
Ordinarily,  there  would  be  exactly  one  element  G  ^  less  than  .’ero  for  each  A,  and 
that  Gf^  would  equal  1.  indicating  the  transition  (of  class  A)  consumes  one  event 
of  class  t  Other  elements  G  .,  j  *  i .  would  bo  either  zero,  indicating  no  involve¬ 
ment  by  the  transition  with  events  of  class  /,  or  positive  integers,  indicating  pro¬ 
duction  by  the  transition  of  G  ^  events  of  class  /  However,  none  of  these  assump¬ 
tions  enters  into  the  mathematics  below,  and  it  can  be  meaningful  to  depart  from 
them 

Using  the  concept  of  event  and  transition  classes,  additional  network  perfor¬ 


mance  parameters  can  be  described  in  terms  of  matrices  M,  R.  S,  and  /  M  is  the 
tim*  taken  by  an  event  of  class  >  to  tra  rerse  link  /  W  is  the  OR)  time  required 

/  IT 

to  receive  an  event  of  class  /  on  node  m  Similarly.  S  is  the  CPU  time  required 

i  m 

to  sent)  an  event  of  class  t  from  node  m  finally,  Is  the  CPU  time  required  to 

process  a  transition  of  class  A  on  node  m  These  matrices  may  often  have  a  fairly 
simple  structure,  for  example,  all  entries  might  have  the  same  value  rr.  3ut 
using  these  matrices  allows  the  representation  of  a  great  deal  of  diversity  among 


the  various  elements  of  a  network,  including  variations  in  the  capacity  of  the  ele¬ 


ments,  and  even  <ariations  in  the  relative  capacity  of  elements  for  operations  of 
different  kinds 


1  f>4 


Chapter  fi  Performance  of  Reference  Tree  Networks 


6.3:  Measures  of  Performance 


[he  purpose  of  engaging  in  this  modeling  effort  is  to  obtain  a  simplified  ahstrac- 
t  in  in  which  the  performance  effects  of  scheduling  strategies  <  an  be  judged 
Before  performance  can  be  measured  and  compared,  though,  it  must  be  defined 
the  plausible  measures  of  performance  are  varied  and  sometimes  antagonistic,  but 
all  are  jiarametei  s  of  particular  execution  histories,  not  of  particular  scheduling 
algorithms  A  scheduling  algorithm,  especially  if  it  contains  any  random  or  pseudo¬ 
random  component,  may  exhibit  different  performance  on  successive  runs,  and  wll 
almost  certainly  have  different  performance  characteristics  when  applied  to 
different  programs  Thus  viur  approach  tc  evaluating  a  scheduling  algorithm  must  he 
to  apply  the  algorithm  to  some  jirogram.  then  examine  the  resulting  execution  his 
tory  to  determine  the  performance  This  approech  to  network  performance  meas¬ 
urement  jiays  one  additional  dividend  it  is  possible  to  define  optimum,  or  mje 
imum,  performance  as  tho  maximum  value  of  the  performance  measure  over  all  pos¬ 
sible  execution  histories  for  the  program  at  hand,  without  respect  to  what,  if  any, 
scheduling  algorithm  could  actually  have  produced  the  winning  history  Then  it  can 
tie  seen  how  far  the  performance  achieved  by  any  particular  scheduling  algorithm 
falls  short  of  the  op'imum  This  may  give  some  indication  of  how  worthwhile  it  is  to 
complicate  a  scheduling  algorithm  in  hopes  of  coming  closer  to  the  optimum  perfor¬ 
mance  level 

Several  plausible  performance  criteria  may  ti  »  derived  from  examination  of  nn 
execution  history 
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•  minimum  resource  us  dye.  The  theory  behind  this  criterion  is  that  the  fewer 
system  resources  (processor  time  and  space,  link  time)  used  in  executing  a 
particular  program,  the  more  are  left  over  for  other  uses  (unspecified  In  the 
model)  Alternatively,  one  may  suppose  that  a  scheduling  algorithm  which  is 
frugal  in  its  use  of  resources  exhibits  greater  potential  to  tie  applied  to 
more  demanding  problems  without  exhausting  the  resources  available  Unfor¬ 
tunately,  incomplete*  use  of  available  re  sources  may  impact  adversely  on 
other  plausible  measures  of  performance,  such  as  response  t.me  Tins  con¬ 
sideration  leads  us.  paradoxically  to  our  next  suggested  performance  meas¬ 
ure 

•  maximum  resource  usage  Tins  criterion  may  be  defended  on  the  grounds 
that  large  degrees  of  resource  usage  are  evidence  that  an  effort  has  been 
made  to  derive*  the  maximum  possible  benefit  from  every  network  resource 
Unfortunately,  high  resource  usage  does  not  necessarily  correspond  to  pro¬ 
ductive  resource  usage,  it  is  entirely  possible  to  consume  resources  in  use¬ 
less  churning  I  urthermoro.  maximum  resource  usage  is  not  desirable  m 
Itself  Rather,  It  Is  conjectured  to  accompany  execution  histories  with  other 
desirable  properties  A  more  direct  approach,  then,  is  to  characterize  those 
properties  directly 

•  minimum  res/*onse  firm*  A  parameter  that  has  more  to  do  wi*h  the  actually 
observed  input/output  behavior  ot  a  system  is  response  time,  but  in  the  con¬ 
text  of  recurring  c  ompula’ions  it'  meaning  needs  to  be  defined  more  pre¬ 
cisely  When  an  event  arrive  at  the  system  anil  passe:  through  several 
transitions,  ultimately  one  or  mor«  output  events,  specifically  traceable  to 
that  input  event,  may  be  caused  The  response  time  for  an  Individual  inpuf 
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event  in  an  execution  history  is  then  the  length  of  time  ttiat  elapses  from 


the  introduction  of  the  Input  event  into  the  system  until  the  last  associated 
output  event  tins  tieen  emitted  from  the  system  If  it  is  possible  tor  some 
Input  events  to  lead  to  no  output  events,  response  time  will  be  a  less  useful 
measure  of  performance,  since  it  must  be  defined  arbitrarily  in  such  cases 
Since  a  particular  response  time  applies  only  to  a  specific  event  in  an  exe¬ 
cution  histoiy,  it  is  possible  to  define  both  an  average  and  a  worst-case 
response  time  over  an  entire  history  Knowing  the  response-time  results  of 
applying  n  s.  heduling  algorithm  to  a  problem  is  indeed  useful,  but  response 
characteristics  are  difficult  to  model  analytically  Ibis  is  because  of  their 
significant  dependence  on  timing  relationships  that  develop  accidentally  dur¬ 
ing  execution,  and  on  the  specific  way  in  which  processors  handle  tasks  (for 
example,  whether  one  task  is  allowed  to  pre-empt  another)  lor  these  rea¬ 
sons,  response  time  can  probably  be  determined  only  by  simulation  or  actual 
execution,  and  the  optimum  possible  response  time  may  be  very  difficult  to 
determine  at  all 

•  minimum  "event  inventory "  An  alternative  parameter,  closely  related  to 
average  response  time,  but  which  may  be  somewhat  more  tractable,  is 
"event  inventory,”  the  number  of  events  that  have  been  introduced  into  the 
system  and  not  yet  discharged  (In  the  sense,  discussed  above,  of  all  asso¬ 
ciated  output  events  having  been  emitted)  As  before,  one  can  talk  about 
average  or  worst-c  ase  event  inventory.  Neither  measure  bears  any  relation 
to  worst-case  response  time,  since  events  are  not  necessarily  discharged  in 
the  order  of  their  introduction,  but  the  average  event  inventory  bears  a  sim¬ 
ple  algebraic  relationship  to  the  average  response  time  Although  It  still 
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resists  exact  treatment,  the  event  inventory  approach  may  facilitate  the 
derivation  of  good  approximations 

•  ihrixjghfHit  We  may  define  the  throughput  of  an  execution  history  as  the 
rate  at  which  input  events  are  discharged  by  the  system  Unless  there  is 
to  be  an  unbounded  buildup  of  events  within  the  system,  tins  rate  will  over 
the  long  term  be;  equal  to  the  rate  at  which  input  events  are  introduced  into 
the  system  Cases  where  unbounded  buildups  of  events  occur  represent 
overloaded  situations  to  which  the  system's  transient  response  may  be 
interesting  but  which  cannot  persist  over  the  long  term  Sint.e,  in  our 
model,  the  rate  at  which  input  events  are  introduced  is  an  independent  vari¬ 
able,  examination  of  a  particular  execution  history  to  determine  throughput 
can  only  yield  a  "yes-no"  answer  yes,  the  system  exhibited  adequate 
throughput  to  prevent  nr.  unbounded  buildup  of  events,  or  no,  it  did  not  It 
is,  however,  possible  to  determine  from  several  execution  histories  gen- 
orated  using  a  particular  scheduling  algorithm  and  different  event  input  rates 
the  maximum  throughput  allowed  by  that  algorithm  ’  What  makes  throughput 
an  especially  interesting  standard  of  comparison  is  that  it  is  also  possible  to 
determine,  for  computations  represented  by  our  model,  the  maximum 
throughput  of  which  a  particular  hardware  configuration  is  capable,  irrespec¬ 
tive  of  the  scheduling  algorithm  used 


"Of  course,  some  algorithms  may  not  be  able  to  meet  certain  requested 
throughputs,  even  whi'e  being  able  to  meet  higher  ones  In  such  cases,  the 
definition  of  "maximum  throughput"  admits  of  some  ambiguity,  but  such  behavior  is 
anomalous,  or  at  least  undesirable,  and  is  best  described  in  terms  of  the  actual 
critical  throughputs  observed,  not  by  attempts  to  define  any  one  "maximum  " 
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On  the  basis  of  the  considerations  discussed  above,  we  select  throughput  as 
the  primary  measure  of  performance  to  use  in  comparisons  Not  only  is  it  analyti¬ 
cally  the  most  convenient,  but  it  captures  well  the  spirit  of  our  broader  endeavor  to 
construct  computer  systems  ot  higher  capacity  The  scheduling  algorithm  with  the 
highest  maximum  throughput  may  not  exhibit  the  best  response  time  at  lower 
throughputs,  but  no  oth*  r  algorithm  can  match  it  when  throughput  is  at  that  max¬ 
imum  High  throughput  it.  also  a  sign  that  inefficient  use  of  resources  due  to 
wasted  motion  15  being  kept  at  a  minimum 

6.4:  Calculating  Maximum  Possible  Throughput 

The  maximum  possible  throughput  of  a  network  is  the  maximum  throughput  of 
which  that  network  is  capable,  regardless  of  the  scheduling  algorithm  used  Tor 
computations  rep.  esentablo  in  our  model,  it  may  bo  calculated,  for  a  specific  net¬ 
work  and  a  specific  type  of  computation,  by  a  linear  programming  procedure 
Numerical  results  may  be  obtained  in  this  fashion,  but  except  in  the  most  elemen¬ 
tary  cases.  <t  is  unfortunately  impossible  to  give  simple  analytical  formulae. 

The  linear  programming  approach  is  based  on  a  kind  ot  network  fiow  model 
which  considers  the  long-term  average  frequencies  of  event  transmissions  over 
links  and  through  nodes,  and  of  occurrences  of  transitions  at  nodes  The  maximum 
attainable  frequencies  are  constrained  by  the  capacities  of  the  various  network 
elements 

In  More  detail,  ’he  linear  programming  model  contains  one  variable  for 

•  each  event  class  for  each  link,  indicating  the  frequency  of  transmissions  of 
events  of  that  class  over  that  link;  we  denote  these  by  e/^,  for  event 
class  I  and  link  j. 
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•  each  transition  class  for  each  node,  indicating  the  frequency  of  execution 

of  transitions  of  that  class  on  that  processor,  those  are  written  as  f. 

Am 

where  the  transition  class  is  A  and  the  node  is  m 

In  the  following  discussion,  we  use  these  and  several  other  variable  names  without 
further  explanation  For  a  summary  of  i  ur  nomenclature,  refer  to  Fable  6  ? 


E 

the  set  of  event  classes 

1 

a  member  of  E 

'o 

the  initial  event  class 

'l 

the  Anal  event  class 

T 

tho  set  of  transition  classes 

A 

an  element  of  T 

L 

the  set  of  links  in  the  network 

1 

an  element  of  L 

L0 

the  set  of  input  links  Lg  t  L 

L! 

the  set  of  output  links  t  L 

N 

the  set  of  nodes  in  the  network 

m 

an  elomont  of  N 

°n 

the  flow  ot  events  of  class  /  over  link  / 

(A  m 

the  frequency  of  transitions  of  class  A  on  node  m 

°,m 

1  Iff  link  /  leaves  node  m.  else  0 

V 

1  iff  link  )  enters  node  m,  else  0 

«« 

the  number  of  events  ot  class  /  generated  by  transition  A 

time  taken  by  an  event  of  class  i  traversing  link 

i 

*tm 

CPU  time  to  receive  an  event  of  class  /  on  node 

m 

S,m 

CPU  time  to  send  an  event  cf  class  /  from  node 

m 

^A  m 

CPU  time  for  a  transition  of  class  A  on  node  m 

Cm 

memory  si/e  at  node  m 

si^e  of  algorithm  implementing  transition  of  class 

A 

Fable  rtf’  Ne'wr.rk  Modeling  Nomenclature 

Fhe  number  of  actually  independent  variables  Is  reduced  by  several  identities 
relating  them  f  or  each  node  m  and  etch  event  clars  /,  there  Is  a  conservation- 
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ot-events  condition 


1(1  O,  )e  ♦  IG  .  f.  -  0  (6.1) 

j'  im  im  1 1  ^  Ik  km 

Furthermore,  for  purposes  of  introducing  events  of  the  initial  class  Iq  and  removing 
events  of  the  final  class  /  ^ .  we  define  a  set  of  in-links  L0  whose  sources  are 
indeterminate  but  whose  destinations  are  nodes,  and  a  corresponding  set  Lj  of 
out-links  I  hen  we  have  the  additional  constraints  that  only  events  of  the  initial 


class  may  flow  over  in-links: 


e  .  =  0,  /  *  and  /  e  Lr 


and  only  events  of  the  final  class  may  flow  over  output  links- 

O'j  =  0,  1*1  j  and  /  *  L  j  (6  3) 

The  constraints  on  the  linear  programming  model  are,  first  of  all,  that  all  vari¬ 
ables  must  be  non-negative.  Events  cannot  "un-flow."  nor  transitions  "un-execute." 
Secondly,  event  flow  over  any  link  j  cannot  exceed  the  capacity  of  the  link 

lM.  e,  .  S  1  (6.4) 


and  computational  activity  (this  includes  execution  of  transitions  as  well  as  proc¬ 
essing  of  messages)  on  any  node  m  must  not  exceed  its  capacity 

II(/  R  *0  S  )e  ,  ♦  IT.  t.  si  (6.5) 

/m  im  jm  im  / /  .  km  km 

'  I  " 

Finally,  subject  to  the  above  identities  and  constraints,  we  wish  to  maximize 


>fL0 

which  is  the  inflow  of  events  o*  the  Initial  class  which  can  be  maintained  over  the 
long  term  without  overloading  any  part  of  the  network  the  values  of  variables 
which  maximize  (6  6)  indicate  optimum  distributions  of  activities  around  the  net- 


Section  6  4  Calculating  Maximum  Possible  Throughput 


6.5:  Memory  Constraint* 

The  model  developed  in  the  previous  section,  although  it  will  tie  our  principal 
source  of  results  concerning  the  theoretical  capacity  of  a  network,  deals  solely 
with  temporal  constraints  on  the  operation  of  the  network  hr  a  real  network,  spa¬ 
tial  constraints  may  very  well  also  play  a  role  In  computations  fitting  our  model, 
space  may  be  used  In  two  ways  for  storage  of  events,  both  while  being  held  at 
nodes  and  while  stored  in  buffers  at  ends  of  links,  and  for  storage  of  the  algorithms 
that  implement  the  transitions  (Reference  tree  overhead  and  temporary  storage 
required  during  actual  execution  of  a  transition  are  taken  into  overhead  ) 

The  linear  programming  model  developed  above,  by  being  insensitive  to  spatial 
constraints,  really  embodies  the  assumption  that  sufficient  space  is  available  at 
every  node  to  buffer  all  events  that  may  need  to  be  buffered  there,  and  to  hold 
algorithms  for  all  transitions  that  may  need  to  be  executed  th'-re  Since  the  model 
concentrates  on  long-term  fk>ws,  it  is  very  difficult  to  determine  anything  about 
buffering  requirements  even  from  examination  of  a  solution  Buffering  requirements 
are  more  a  p-operty  of  the  microstructuro  of  operations  on  each  processor,  a  vari¬ 
able  deliberately  abstracted  out  in  the  course  of  building  our  mode!  In  many 
cases,  though  eve-os  may  be  a*sumed  to  be  relatively  small,  and  the  buffering 
requirements  relatively  modest,  or  a!  least  relatively  unaflected  by  changes  in 
event  distribution.  In  such  cases,  the  storage  of  algorithms  may  he  the  main  contri¬ 
butor  to  memory  use 

A  simple  assumption  about  memory  use  i?  that  If  a  transition  is  executed  at  all 
on  a  processor,  however  infrequently,  a  copy  of  the  corresponding  algorithm  must 
reside  on  that  processor  Then  the  memory  used  at  node  m  is 

JVl  (,Am>  <6  7> 

where  P^  is  the  sire  of  the  algorithm  implementing  transitions  'it  class  A,  and 
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is  the  unit  step  function 


jo  if  *  i  0 

U1U)  “  \l  if  *  >0 


(6.8) 


Note  that  expression  (6V)  Is  nonlinear  in  and  hence  cannot  be  used  in  either  a 
constraint  or  an  optimality  criterion  in  linear  programming  Be  that  as  it  may,  a  more 
serious  question  is  how  memory  usage  ought  to  be  accounted  for  in  any  model, 
linear  or  not  Should  seme  weighted  average  of  throughput  and  memory  usage  be 
used  as  the  optimality  criterion,  rattier  than  throughput  alone?  Rut  what  is  the 
point,  given  sufficient  memory  in  the  system,  of  settling  for  lower  throughput  simply 
bocause  it  makes  less  intensive  use  of  memory  the  system  already  has?  One 
Justification  for  so  doing  is  the  existence  in  the  system  of  activities  other  than 
that  being  modeled.  If  the  computation  being  modeled  takes  up  less  space,  then 
more  space  is  free  for  other  uses  But  why  account  for  the  spatial  Impact  of  other 
activities  when  no  such  accounting  is  made  of  their  temporal  impact?  On  the 
whole,  it  is  of  some  interest  to  observe  the  memory  requirements  of  the  optimal 
(maximum  throughput)  solution  or.  If  there  are  several,  the  optimal  solution  requiring 
the  least  memory,  but  it  Is  difficult  to  justify  proclaiming  the  superiority  of  a  lower- 
throughput  solution  simply  on  the  grounds  that  it  uses  less  memory 

A  different  tack  is  to  treat  memory  usage  not  as  part  of  any  optimality  cri¬ 
terion,  but  as  a  constraint  on  possible  solutions  Borrowing  from  equation  (0  7),  a 
new  constraint  of  the  form 


(6  9) 


could  be  added  for  each  node  m,  where  C  is  the  memory  capacity  at  node  m. 
The  resulting  problem  is  no  longer  a  linear  programming  program,  but  may  still  be 
solved  by  combinatorial  means.  This  approach  at  least  begins  to  deal  with  the 
reductions  in  network  capacity  that  may  be  forced  by  space  constraints  on  the  dis¬ 
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tribution  of  algorithms  (we  continue  to  ignore  space  requirements  for  buffering). 
Such  space  constraints  will  only  be  significant,  however,  if  the  size  of  algorithms  is 
of  the  same  order  of  magnitude  as  the  memory  capacities  of  processors  for  com 
potations  accurately  representable  by  our  simple  model,  this  is  unlikely  to  be  the 
case  furthermore,  the  purpose  of  constructing  this  model  is  to  determine  the 
optimum  performance,  for  comparison  with  that  exhibited  by  various  scheduling  algo¬ 
rithms  Incorporating  into  these  algorithms  consciousness  of  fairly  severe  space 
constraints  adds  a  whole  new  level  if  complication  to  their  construction  and 
analysis 

for  those  reasons,  memory  usage  will  be  of  relatively  little  interest  to  us 
There  is.  however,  room  to  wonder  whether  our  step-function  approach  to  measuring 
memory  usage  is  always  the  most  accurate  At  a  processor  where  some  transitions 
are  being  executed  only  relatively  infrequently,  and  neighboring  network  elements 
are  not  fully  loaded,  a  dynamic  solution  in  which  algorithms  are  continually  moved 
back  and  forth  may  actually  further  economize  memory  usage  without  ri'diicing  net¬ 
work  throughput  lhis  may  cause  tolal  memory  usage  to  be  lower  than  given  by 
equation  (6  7)  On  the  other  hand,  (6  7)  certainly  provides  an  upper  bound  on 
memory  requirements,  as  long  as  buffering  requirements  continue  to  be  regarded  nr. 
negligible 

6.0:  Discussion 

Simple  and  limited  though  our  linear  programming  model  may  bo.  it  can  be  used 
to  gain  understanding  of  some  important  situations,  and  perhaps  explode  a  few 
preconceptions  one  might  be  tempted  to  form  regai ding  the  performance  of  refer¬ 
ence  tree  networks  One  such  preconception  is  that  throughput  of  a  reference 
tree  network,  calculated  as  described  earlier  in  this  chapter,  can  he  made 


164 


Chapter  6:  Performance  of  Reference  Tree  Networks 


arbitrarily  large  by  just  adding  more  processors  As  long  as  communication  costs 
are  nonzero,  this  is  not  true  Also  untrue,  as  long  as  communication  has  a  cost,  is 
the  thesis  that  adding  more  processors  will  cause  a  linear  increase  in  throughput 
These  statements  should  be  qualified  somewhat  increasing  the  number  of  proces¬ 
sors  will  always  result  in  a  linear  increase  in  network  capacty.  but  if  events 
emanate  from  a  fixed  number  of  sources,  more  and  more  of  that  capacity  will  be 
eaten  up  in  communication  overhead  as  the  network  grows.  If,  on  the  other  hand, 
the  number  of  event  sources  increases  with  the  size  of  the  network,  this  effect 
can  be  avoided 

6.6. 1  :  Performance  of  Some  Simple  Topologies 

A  few  examples  w,"  help  illustrate  this  point,  and  also  give  some  context  f-v 
evaluating  MuNet  performance  results  to  bo  presented  later  Consider  the  simplest 
possible  computation,  in  which  there  are  only  two  event  classes  the  initial  event 
class  and  the  final  event  class  There  is  only  one  kind  of  transition,  which  converts 
an  event  of  the  initial  cluss  into  an  event  of  the  final  class  As  far  as  physical 
properties  of  the  network  are  concerned,  we  assume  that  all  processors  and  links 
are  Identical  (r  e.,  have  the  same  performance  parameters)  We  further  assume 
that  the  bandwidth  of  links  does  not  constrain  system  throughput  This  is 
equivale' t  to  positing  that  the  1  PU  time  to  process  incoming  and  outgoing  everts  is 
greater  than  the  link  time  required  for  them  transmission,  something  which  is  true  of 
the  MuNe»  and  a  plausible  hypothecs  about  many  other  systems  Finally,  we 
assume  (hat  the  CPU  send  an  .  receive  times  for  all  events  are  equal  to  the  con¬ 
stant  cli  t-  R  -  S  -  cl  and  tti.it  tne  time  to  execute  a  transition  on  any  proc- 
essor  is  f  (/.e.,  T -  f) 

Consider  then  the  p«  formince  of  .1  "pipeline"  i  f  N  processors,  as  shown  in 


Section  6.6.1  Performance  of  Some  Simple  Topologies 


165 


Figure  €3.3:  A  pipeline  topology 


I  igure  6.3,  where  events  of  the  initial  class  are  presented  at  processor  1  and 
events  of  the  hrial  class  are  extracted  from  processor  N  Our  linear  programming 
analysis  gives  as  the  throughput  #  of  this  pipeline 

t  -  —N — ■  (6  10) 

f » ?cN 

The  numerator  in  this  expression  gives  the  total  number  of  units  of  CPU  time  avail¬ 
able  per  unit  of  real  time,  while  the  denominator  indicates  the  total  number  of  units 
of  CPU  t.me  needed  to  process  each  incoming  event  t  units  of  time  to  actually 
execute  the  transition  that  converts  it  from  an  initial  event  to  a  final  event,  and  ?c 
time  units  per  processor  to  forward  the  event  along  from  the  beginning  to  the  end 
of  the  pipeline  As  N  becomes  large,  the  throughput  approaches  an  asymptote  of 

'  ,  as  each  processor  spends  a  higher  and  higher  proportion  of  its  time  simply  for- 
?c 

warding  events  Thus  adding  unbounded  numbers  of  processors  does  not  result  in 
unbounded  increases  in  throughput  It  is  possible  to  giaph  equation  (6.10)  as 
shown  in  figure  6.4 

In  irder  to  avoid  the  performance  deg  adnt  on  that  comes  with  having  to  send 
every  event  through  a  long  pipeline,  whi.e  still  hnvinq  only  one  source  and  sink  of 
events,  we  migh*  be  tempted  to  try  a  "bushier”  structure  such  as  that  in  Figure 
6  6*  The  linear  programming  model  now  gives  a  throughput  of 

*This  topology  may  lie  construed  as  violating  the  "limited  number  of  neighbors" 
condition  of  reference  tree  network  topolog.es,  but  this  limitation  is  less  important 
for  our  purposes  than  the  performance  limitations  of  the  topology 
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?. 


Number  of  processors  N  > 


I  -  time  to  execute  n  transition  on  a  processor 
r  -  CPU  time  to  send  or  receive  an  event  on  a  processor 

Figure  6  4  Throughput  of  the  pipeline  topology  shown  in  figure  6  3 


Figure  Bh  A  parallel  topology 
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winch  increases  linearly  with  N  up  to  the  critical  throughput  Uelow  this  critical 

2c 

throughput,  an  analysis  similar  to  that  in  our  previous  case  holds  N*2  units  of  CPU 
time  are  available  per  unit  of  real  time,  and  each  event  takes  lie  time  units  tor 
communication  (to  be  received  and  sent  at  each  of  three  processors)  and  I  time 
units  to  be  converted  from  an  initial  event  to  a  final  event  Hut  the  throughput  can¬ 
not  exceed  '  because  at  this  point  the  left-hand  and  right-hand  processors, 
2c 

through  which  all  events  must  pass,  become  fully  loaded  with  communications  and 
cannot  handle  any  additional  events 

In  many  cases,  such  as  the  actual  performance  tests  that  were  run  on  the 
MuNet.  events  of  the  final  class  must  be  taken  out  of  the  system  at  the  same 
processor  where  events  of  the  initial  class  are  introduced  In  other  words,  th« 
answers  must  appear  at  the  same  place  where  the  problems  were  posed  Thus  the 
evaluation  nf  MuNet  performance  results  is  more  directly  related  to  two  topologies 
we  now  consider  The  ftrsf  of  these,  shown  in  figure  6  0,  is  similar  to  the  parallel 
topology  of  figure  (>  b  ext  opt  that  the  source  and  -.ink  for  events  is  on  the  saru 
processor 


figure  6  6  Parallel  topology  with  common  source  and  sink 
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lhe  linear  programming  model  gives  for  the  throughput  of  this  network 


9  - 


1 

I  «Pc 


1  ‘(f  Pc  )min  |  1  ,  ^ 
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which  increases  linearly  with  N  until  it  roaches  a  maximum  value  of  9  -  -  at 

4c 


N  -  1  <  ^  Numbers  representative  of  the  MuNet  applications  to  bo  discussed 

4c  ? 

later  are  I  :  OS  seconds  and  i  =  30  to  70  ms,  giving  a  maximum  throughput  of  4 
to  ft  tasks  per  second,  first  reached  for  N  between  7  and  3  Obviously  this  number 

is  strongly  influenced  by  ttie  ratio  *  that  characterizes  a  particular  application,  on 

c 

the  MuNet.  estimates  of  c  are  particularly  unreliable,  since  communication  overhead 
scorns  to  depend  in  a  significantly  nonlinear  way  on  the  amount  of  communication 
and  on  the  kinds  of  other  activities  occurring  on  a  processor  and  its  neighbors 


••• 

Figure  fi  7  "Stack"  topology 


A  final  family  of  topologies,  interesting  because  it  was  also  the  subject  of 
actual  MuNet  experiments,  and  interesting  as  well  because  it  is  a  simple  example 
of  a  family  of  topologies  whose  maximum  possible  throughput  has  no  simple  closed 
form,  is  the  family  of  "stack”  topologies  illustrated  in  figure  fi  7  The  throughput 
9(N)  of  a  stack  topology  of  length  N  is  given  by  the  recurrence  relation 


9(N  •  1  )  -  m.n  1 

t 4r  f  »?c  ) 


90)  - 


1 


(0  13) 


t  •  Pc 

A  plot  of  9<S)  using  the  parameter  values  f  =  Oft  seconds  and  c  =  30  ms  Is  shown 
In  F  iguro  fi  ft 
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N  =  10  N  *  P0 


N  -----  > 

0  b  second1}  to  execute  a  transition  on  n  processor 

30  ms  of  CPU  time  to  send  or  receive  an  event  at  a  processor 

figure  0  H  Performance  of  the  stack  topology  shown  in  Figure  Ci  i' 

8.8.2:  Assignment  of  Events  to  Processors 

Other  applications  of  the  linear  programming  model  are  to  computations  more 
complex  than  the  very  simple  case  we  have  been  considering  When  there  are 
several  classes  of  evonts,  the  strategy  Issue  arises  of  which  kinds  of  events  to 
execute  where  This  decision  is  not  alwa>  s  as  simple  as  It  might  seem  For  exam¬ 
ple.  consider  the  computation  of  the  function  f(g(*))  of  some  stream  of  input 
values  r  Such  n  computation  might  be  performed  in  two  stops  and  involve  three 
classes  of  events  an  initial  class  containing  input  values  *.  an  intermediate  class 
containing  partial  results  g(  * ).  and  a  final  class  contacting  results  fTgix)) 

-  . -*u} . -> 

figure  fi  T1  T  wo-processor  pipeline  topology 
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If  this  computation  wme  performed  on  a  t wo-proce-.sor  pipeline  topology  such 


as  that  shown  in  I  iguro  0.9,  it  would  seem  appropriate  to  make  the  "natural" 
correspondence  between  the  pipelined  computation  and  ttie  pipeline  topology  by 
doing  all  computations  of  y  on  processor  1  and  all  computations  of  I  on  processor 
2  Such  an  assignment  is  ai  tually  beM.  however,  only  under  fairly  limited  sets  of 
circumstam  es  t  irst  of  all,  it  will  not  generally  be  the  case  that  this  assignment 
will  result  in  a  balanced  load  on  the  system  only  for  very  specific  values  of  com¬ 
munication  and  computation  costs  will  performing  computations  of  y  on  processor  1 
at  some  rate  •  keep  that  proi  essor  tully  utilized  while  computations  of  1  on  proces¬ 
sor  2  occurring  at  the  same  rate  #  are  also  keeping  thu  processor  fully  utilized 
More  likely,  whichever  processor  has  been  a-  jned  the  computation  that  involves 
lower  computation  or  communication  costs  will  have  some  idle  time,  meaning  ttiat 
system  throughput  could  be  increased  by  shifting  to  that  processor  some  ot  the 
load  on  the  other  processor  Thus  if  the  computation  of  y  were,  say,  only  half  as 
expensive  as  that  of  f,  throughput  greater  than  that  exhibited  by  the  "obvious" 
pipeline  would  be  achieved  by  a  solution  in  which  processor  1  performed  all  compu¬ 
tations  of  y  and  also  some  of  I.  while  processor  2  performed  the  remaining  compu¬ 
tations  of  f 

Beyo-id  this  minor  load-balancing  adjustment,  however,  lies  room  for  questioning 
the  whole  idea  of  pipe'ining  the  computation  at  all  An  alternative  is  to  have  each 
processor  perform  equal  numbers  of  computations  of  1  and  y,  never  passing  any 
Intermediate  results  y( » )  between  processors  Only  events  of  the  initial  and  final 
classes  would  then  tie  passed  between  processors  This  approach  will  be  inferior 
If  the  intermediate  events  are  less  expensive  to  pass  between  processors  than 
the  initial  or  final  events,  for  then  it  will  pay  to  convert  initial  events  as  quickly  as 
possible  to  the  more  cheaply  communicated  intermediate  form,  and  to  put  off  for  as 
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lony  as  possible  the  conversion  to  the  final  form  that  is,  it  will  pay  to  use  the 
pipelined  approach  On  the  other  hand,  If  the  intermediate  events  are  more  expen¬ 
sive  to  communicate  than  the  initial  or  final  events  (a  common  situation  situation  In 
programs  which  build  up  a  large  collection  of  partial  results  before  disgorging  an 
answer),  then  the  highest  throughput  will  be  attained  if  intermediate  events  are 
never  communicated,  but  are  consumed  as  soon  as  they  are  produced 

Thus  the  unmodified  "natural"  pipeline  arrangement  will  very  rarely  yield  the 
optimal  throughput  In  most  cases,  the  only  reason  that  would  dictate  such  on 
arrangement  would  be  memory  constraints  making  it  impossible  to  fit  the  algorithms 
for  t  and  y  both  on  the  same  processor  Although  this  might  sometimes  bo  a  real 
problem,  it  is  not  likely  to  be  on  overriding  concern  in  general 

The  observations  made  in  this  section  generalise  in  the  obvious  way  to  longer 
pipelines  The  linear  programming  model  remains  able  not  only  to  give  the  maximum 
possible  throughput  of  such  configurations,  but  also  to  indicate  the  distributions  of 
events  which  will  bring  about  the  maximum  possible  throughput  Perhaps  more 
important  than  this  capability,  however,  is  the  lesson  thct  the  distributions  that 
seem  the  most  natural  are  not  always  the  best,  and  that  the  look  of  a  distribution 
which  is  optimal  may  be  profoundly  influenced  by  small  changes  in  the  relative 
costs  of  various  operations  A  lesson  for  the  development  of  scheduling  strategies 
is  that  appealing  heuristics  such  as  "find  and  exploit  any  natural  pipeline  structures 
in  the  object  program"  should  he  approached  with  extreme  cant  on  A  scheme  that 
is  likely  to  be  more  generally  satisfactory  ought  to  adjust  its  behavior  according  to 
the  run-lime  load  effects  produced  by  previous  decisions,  or  else  ought  to  rely  on 
the  results  of  a  more  thorough  analysis  of  the  program  to  be  scheduled 
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6.7:  Summary 


A  model  has  boon  developed  for  the  performance  of  reference  tree  networks 
executing  certain  simple  computations  of  a  recurring  nature  Applications  that 
would  fall  into  this  class  include  some  kinds  of  real-time  data  acquisition,  along  with 
periodic  repeated  computations  such  as  would  be  required,  for  example,  to  update 
a  moving  picture  of  a  three-dimensional  scune  Using  linear  programming  methods,  it 
is  possible  to  determine  the  max  mum  possible  throughput  for  such  computations  of 
a  particular  network  This  throughput  can  then  be  used  as  a  standard  for  compari¬ 
son  with  the  throughput  achieved  by  particular  scheduling  algorithms 

The  linear  programming  model  can  also  be  u?  ed  to  explore  the  inherent  perfor¬ 
mance  limitations  of  various  topologies,  and  to  look  at  strategies  for  assigning 
events  to  processors  The  discussion  of  these  topics  at  the  end  cf  this  chapter  :s 
not  intended  to  convey  the  impression  that  reference  tree  networks  are  not  scal¬ 
able  Tor  one  thing,  the  point  at  which  the  limitations  examined  become  manifest 
depends  sir  ngly  on  the  ratio  of  computation  to  communication  speed  at  each  node 
More  importantly,  the  examples  in  Section  R  fi  are  not  designed  to  imply  that  refer¬ 
ence  tree  network  capacity  increases  less  than  linearly  in  the  si/o  of  the  network 
The  speed  of  performing  any  one  set  of  calculations  will  at  some  point  stop 
increasing  as  more  processors  are  added,  because  at  some  point  communication 
costs  will  dominate  the  extra  computation  power  available,  however,  a  larger  net 
work  will  still  have  increased  capacity  for  ndci itional  concurrent  tasks 


Section  6V  Summary 


1  73 


Chapter  7:  Scheduling  Strategies  for  Reference  Tree  Networks 


Ihe  term  "scheduling  strategies"  is  perhaps  something  of  a  misnomer  tor  the 
subject  of  this  chapter,  which  is  little  concerned  with  the  scheduling  of  events  in 
time,  but  much  more  concerned  with  the  distribution  of  objects  and  events  in 
space,  /.e.,  their  allocation  among  the  various  nodes  of  the  network  Ihe  former 
subject  is  not  wholly  without  interest  the  treatment  try  the  system  of  tasks  with 
differing  priorities,  or  the  response  of  the  system  to  tasks  with  real  time  con¬ 
straints,  would  be  important  issues  in  many  cases  However,  spatial  scheduling 
decisions  crucially  affect  the  efficiency  of  tfie  system  by  determining  the  amount  ot 
internode  communication  that  will  be  necessary  in  the  course  of  executing  a  pro¬ 
gram  Ihe  amount  of  this  overhead  can  have  a  profound  impact  on  the  ability  of 
the  system  to  meet  real-time  demands,  but  is  of  interest  even  to  the  user  whose 
only  desire  is  that  the  system  oxecute  his  program  in  a  reasonably  expeditious  and 
fair  manner 

An  interesting  attribute  of  scheduling  strategies  that  sets  them  apart  from  the 
rest  of  the  material  presented  in  this  thesis  is  their  discretionary  nature.  Up  to 
this  point,  actions  performed  by  the  system  have  been  prescribed  by  algorithms  for 
conflict  resolution,  reference  tree  maintenance,  etc.  These  algorithms  are  applied 
mechanistically,  after  having  been  invoked  either  by  some  part  of  a  user’s  program, 
or  by  desires  whose  origin  has  until  now  remained  unspecified,  such  as  a  desire  on 
the  part  of  a  processor  to  ootam  locally  a  copy  of  a  text.  The  origin  of  these 
desires,  which  are  not  expressed  directly  by  the  user,  is  the  mtwork  scheduling 
strategy  For  example,  if  the  scheduling  strategy  determines  that  an  event  is  to 
be  executed  on  a  particular  processor,  and  that  event  requires  a  copy  of  a  certain 
object  text,  then  it  v'ill  become  necessary  to  obtain  a  copy  of  that  text  on  that 
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In  the  context  of  a  reference  tree  network,  the  discretion  that  may  be  exer¬ 
cised  t>y  scheduling  strategies  is  extremely  broad  VIM  programs  contain  no  expli¬ 
cit  directions  for  allocation  of  events  and  objects  to  processors,  and  the  reference 
tree  algorithms  will  support,  at  varying  levels  of  efficiency,  any  distribution  whatso¬ 
ever  furthermore,  moving  events  or  objects  is  sufficiently  inexpensive  that  It  is 
practical  whenever  there  is  an  indication  that  some  benefit  might  be  gamed 
thereby 

One  degree  of  freedom  open  to  a  scheduling  strategy  is  the  choice  between 
demand-driven  and  what  might  bo  called  notice-driven  behavior  Any  scheduling 
strategy  will  have  to  face  situations  that  demand  some  action,  For  example,  an 
executing  event  may  request  access  to  some  text  not  rurrr*nlly  locally  available 
In  such  a  case,  there  are  several  plausible  courses  of  action  a  request  can  be 
made  to  bring  a  copy  of  the  desired  text  to  the  current  processor  (so  that  the 
event  can  resume  execution),  the  event  can  be  sent  to  a  processor  that  already 
has  a  copy  of  the  text,  or  some  compromise  solution  can  he  adopted  Whatever 
decision  is  marie  affects  the  distribution  of  events  and  objects  and  is  properly 
termed  a  scheduling  decision  A  scheduling  strategy  might  also  operate  in  a  more 
contemplative  modi',  however,  observing  the  operation  of  the  system  and  making 
adjustments  that,  although  not  demanded  by  any  immediately  critical  situation,  will 
help  the  system  operate  in  a  more  efficient  manner  in  the  future.  Fxamples  of  such 
"notice -drivn"  scheduling  behavior  include  processor  and  memory  load  balancirg. 
grouping  of  related  objects  on  nearby  processors,  and  the  like  This  chapter  will 
concentrate  mainly  on  notice-driven  strategies,  for  it  is  here  that  the  choices  are 
widest  and  the  opportunities  greatest 
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7.1:  The  Diffusion  Strategy 


The  simplest  imaginable  load-balancing  strategy  that  operates  strictly  on  a 
local  basis  is  a  "diffusion"  strategy  in  which  load  is  continually  transferred  over 
each  link  from  the  more  loaded  end  to  the  less  loaded  end  Under  this  strategy, 
any  concentration  of  load  at  a  particular  point  will  gradually  flatten  and  spread  out 
with  time,  much  as  impurities  diffuse  through  a  crystal  lattice  This  strategy  can  be 
applied  both  to  events  (primarily  to  balance  processing  load)  and  objects  (primarily 
to  balance  memory  load)  The  strategy  is  appealing  for  its  simplicity  and  elegance, 
but  the  applicability  of  the  simplest  version  is  limited  by  the  fact  That  it  'gnores 
information  about  relationships  between  events  and  objects  which  could  be  used  to 
make  better  scheduling  decisions  Nevertheless,  it  does  begin  to  address  the 
issuo  of  bringing  into  play  as  many  as  possible  of  the  resources  of  a  network,  and 
can  be  modified  to  take  into  account  some  of  the  information  that  the  simple  ver¬ 
sion  ignores  The  diffusion  strategy  is  also  of  interest  because  it  is  the  current 
scheduling  strategy  on  the  MuNet,  and  thus  some  preliminary  performance  results 
can  be  reported 

The  current  diffusion  strategy  for  events  on  the  MuNet  incorporates  a  parame¬ 
ter  known,  for  historical  reasons,  as  OrutlGf  The  eagerness  of  events  to  diffuse 
throughout  the  MuNet  is  curbed  by  the  requirement  that,  in  order  for  an  event  to 
diffuse  from  a  processor  to  its  neighbor,  the  number  of  active  events  on  the  proc¬ 
essor  must  exceed  that  on  the  neighbor  by  at  least  QFUOGf  There  are  two  rea¬ 
sons  for  do-ng  this  One  is  to  enhance  the  s*nb  'it y  of  the  system,  and  cut  down  on 
redundant  communication,  by  trying  to  ensure  that  once  an  event  has  diffused  to  a 
new  processor,  it  will  not  immediately  diffuse  back  The  other  is  that  diffusing 
events  across  a  wider  set  of  processors  will,  In  general,  increase  the  amount  of 
commune  ation  that  must  occur  A  particular  \  alue  of  QFUDCF  reflects  a  certain 
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tradeoff  between  the  benefits  of  using  more  of  the  system  resources  and  the  extra 
communication  costs  incurred. 

7.1.1:  Tests  for  the  Diffusion  Strategy 

It  is  easy  to  constiuct  examples  where  the  simple  diffusion  strategy  will  exhi¬ 
bit  very  poor  performance  the  more  structured  the  relationships  among  the  events 
and  objects  that  make  up  a  computation,  the  more  likely  diffusion  is  to  make  "hash" 
out  of  that  structure  and  stretch  communication  lines  to  unfortunate  lengths 
Indeed,  many  programs  ttiat  would  actually  be  wr  iten  (assuming  they  contain 
enough  parallelism  to  be  able  to  benefit  from  multiprocessing  In  the  first  place) 
would  probably  suffer  under  the  diffusion  strategy  This  section  includes  results 
from  some  t**sts  contrived  to  show  the  diffusion  strategy  in  the  best  possible  light, 
to  give  an  indication  of  what  is  possible  and  of  what  may  be  the  ultimate  limitations 
of  the  approach  While  the  actual  computations  performed  were  rather  artificial,  an 
attempt  will  be  made  to  relate  these  to  actual  tasks  that  one  miyht  carry  out  using 
a  reference  tree  network 

In  the  first  test,  which  we  shall  call  the  "asynchronous"  test,  a  controlled 
number  of  parallel  tasks  is  introduced  into  the  network,  such  that  each  task  com¬ 
putes  for  a  fixed  period  (this  is  a  fixed  amount  of  CPU  time,  not  a  fixed  period  of 
real  time),  notifies  a  metering  actor  of  its  completion,  and  then  lestarts  execution 
from  flu*  beginning  of  the  task  The  tasks  are  simple  timing  loops  not  performing 
any  useful  computation,  but  the  scenario  is  simi'ar  to  that  of  a  system  periodically 
accepting  packets  of  input  data,  processing  them  In  some  fashion,  and  emitting  out¬ 
put  packets 

To  form  an  exact  analogy  with  our  test,  a  pocket-processing  application  would 
have  to  have  the  property  that  a  new  input  packet  is  received  every  time  an 
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output  packet  is  emitted  Then  the  "degree  of  parallelism,"  or  total  number  of  input 
packets  currently  undergoing  processing,  always  remains  constant  Perhaps  a  more 
realistic  scenario  would  be  to  assume  that  input  packets  arrived  at  regular  inter¬ 
vals,  Irrespective  of  the  rate  of  generation  of  output  (jackets,  lhis  was  not  done 
on  the  MuNet  due  to  the  difficulty  of  generating  tasks  at  precisely  timed  moments 
However,  a  glance  at  the  graphs  presented  helo.v  should  convince  the  reader  that 
if  packets  are  presented  at  regular  intervals  not  shorter  than  the  average  interval 
between  task  completions  in  our  test  (for  the  particular  degree  of  parallelism  that 
is  of  interest),  then  the  MuNet  shout  )  have  adequate  throughput  to  handle  the  job 
Another  test  (the  "synchronous"  test)  conducted  on  the  MuNet  is  closely 
related  to  this  one,  except  that  after  the  controlled  number  of  tasks  is  introduced 
Into  the  system,  all  must  report  their  completion  to  the  metering  actor,  after  which 
all  will  be  restarted  in  unison  A  real-world  analogy  for  this  test  might  bo  found  in  a 
graohic  display  system,  where  at  intervals  (for  example,  when  a  viewpoint 
changes)  it  might  be  desired  to  recalculate  the  position  of  each  point  and  line  on 
the  screen  Whereas  the  asynchronous  test  reveals  on'y  the  average  delay 
between  the  initiation  of  a  task  and  its  completion,  the  synchronous  tost  is  sensi¬ 
tive  to  Aforsl-cast'  behavior,  and  will  perform  more  poorly  than  the  asynchronous 
one  if  the  notwoik  takes  anomalously  long  times  to  complete  some  of  the  tasks 

A  third  test,  conducted  under  somewhat  less  rigorously  controlled  cir¬ 
cumstances.  was  of  a  program  to  compute,  in  a  parallel  fashion,  solutions  to  the 
classical  "eight  gn  >ens"  problem  The  p'oblem  Is  to  find  placements  for  eight 
gi  eens  or  a  standarc  chessboard  such  that  no  gueen  could  in  one  move  capture 
any  of  the  others.  It  is  clear,  from  the  rules  of  chess,  that  every  gueen  must  be 
placed  in  a  different  file,  or  column,  and  that  one  gueen  must  be  placed  in  each  file 
Thus  a  standard  algorithm  for  solving  the  problem  is  to  tentatively  place  .>  gueen  in 
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one  of  the  squares  of  the  first  file,  and  then  continue  placing  queens  in  subsequent 


files  in  positions  where  they  could  not  tie  captured  by  any  queens  already  placed 
If  it  is  found  impossible  to  place  a  queen  m  some  file,  then  the  algorithm  must  back¬ 
track.  and  find  a  different,  previously  untried  arrangement  of  the  queens  already 
placed  fhe  algorithm  in  effect  searches  a  tree  of  possibilities,  where  each  inter¬ 
nal  node  corresponds  to  some  partial  queen  placement  Lach  such  node  has  eight 
sons  corresponding  to  the  eight  possible  placements  of  a  queen  in  tile  next  file 
leaf  nodes  represent  potential  solutions  in  which  all  eight  queens  have  been 
placed  A  tree-pruning  heuristic  in  the  algorithm  causes  backtracking  when  it  is 
clear  that  no  leaf  node  under  the  current  node  could  possibly  be  a  solution 

The  program  tested  on  the  MuNot  explores  the  same  tree,  but  all  of  a  node's 
sons  that  cannot  immediately  be  pruned  are  explored  in  parallel,  leading  to  a  max¬ 
imum  of  several  hundred  concurrent  tasks  Fach  internal  node  under  which  active 
exploration  is  proceeding  has  an  associated  object  recording  the  number  of  active 
sons  of  that  node,  and  the  number  of  solutions  that  have  been  found  so  far  The 
solution  count  is  passed  back  up  the  tree  as  computations  finish,  until  when  all  com¬ 
putations  cense  the  root  node  has  a  record  of  the  total  number  of  solutions  found 
Thus  the  eight  queen:  proqr  im  does  have  some  structure  of  interroferencing 
objects,  and  ns  a  result  we  would  expect  the  simple  diffusion  strategy  to  bo  less 
kind  to  it  and  lead  to  higher  communication  costs  than  in  our  previous  tests  An 
additional  factor  pointing  m  this  direction  is  that  .  alculations  in  the  eight  queens 
program  arc-  not  artificially  prolonged,  as  in  our  previous  tests,  thus  the  ratio  of 
communication  time  to  computing  time  should  he  higher 
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7.1.2:  MuNet  Performance  Peculiarities 

I  ho  reader  will  be  able  to  make  a  more  educated  assessment  of  MuNet  perfor¬ 
mance  results  if  a  few  quirks  in  the  operation  of  the  MuNet  are  explained  first  In 
the  MuNet,  being  attached  to  a  communication  link  is  an  item  of  significant  CPU  time 
overhead  for  a  processor,  whether  any  objects  or  events  are  being  communicated 
over  the  link  or  not  this  is  because  cf  a  steady  dialogue  of  sta’us  and  ack¬ 
nowledgment  messages  maintained  over  each  link  by  the  processors  at  its  ends, 
plus  some  reference  tree  protocol  messages  triggered  by  garbage  collection 
activity  Thus,  all  other  things  being  equal,  adding  a  link  to  a  system  reduces  the 
amount  of  available  CPU  time  in  t fit*  system  Comparing  one-processor  topologies  to 
two-processor  topologies  is  especially  hazardous,  for  on  the  one-proressor  net¬ 
works  all  the  CPU  time  can  be  devoted  to  event  execution,  while  on  the  two-proc¬ 
essor  versions  not  only  does  the  diffusion  strategy  have  its  first  opportunity  to  mix 
things  up,  but  also  the  overhead  associated  with  the  link  is  felt  for  the  first  time 

Amusingly  enough,  if  a  processor  is  sufficiently  busy  executing  events,  the  load 
it  causes  its  neighbors  usually  drops'  Tins  ,s  because,  with  a  surplus  of  events  to 
execute,  less  time  is  devoted  to  garbage  collection  and  generation  of  status  infor¬ 
mation  Thus  the  performance  statistics  show  a  reduction  in  communication  link 
overhead  for  heavily  loaded  MuNets 

Neither  of  these  performance  juirkr  is  an  inherent  property  of  the  reference 
tree  network  architecture,  the  cause  is  the  slowness  of  the  LSI-11  microproces¬ 
sors  in  the  MuNet,  combined  witti  tho  simplicity  of  the  link  haidware  which,  as  a 
result,  requites  a  fan  amount  of  attei  tion  from  Hie  processor  The  quirks  are  dis¬ 
cussed  here  only  so  that  the  reader  can  discount  their  effects,  to  the  extent  that 
he  feels  is  justified.  In  examining  the  following  performance  results 
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7.1.3:  MuNet  Performance  Results 


All  MuNet  performance  tests  includod  a  special  metering  actor  which  recorded 
the  start  and  finish  times  tor  task  executions  I  or  a  variety  of  reasons,  many  of 
which  are  uninteresting  from  the  point  ot  view  of  this  thesis,  the  metering  actor 
used  a  special  MuNet  facility  to  "wire  itself  down"  to  a  particular  processor  (this 
facility  is  described  in  Appendix  A)  Thus  all  new  tasks  were  created  on  this  proc¬ 
essor,  and  all  completed  tasks  ultimately  had  to  report  back  to  it  One  dividend  of 
this  approach  Is  that  it  makes  possible  a  fairly  direct  comparison  to  the  linear  pro¬ 
gramming  model,  assuming  one  input  link  and  one  output  link  each  attactied  to  the 
processor  containing  the  metering  actor  Another  payoff  Is  that  the  metering  actor 
can  record  the  proportion  of  time  spent  on  its  processor  tor  event  execution,  com¬ 
munication,  and  <dhor  overhead  (also  getting  a  simultaneous  picture  of  the  state  of 
other  processors  is  harder)  This  data  helps  explain  some  of  the  performance 
results  recorded,  and  also  provides  the  basis  for  crude  estimates  of  performance 
parameters  for  comparisons  with  the  linear  programming  model 

The  execution  times  for  tasks  used  in  those  tests  are  quite  long  In  part,  this 
is  due  to  the  slowness  of  the  I  SI- 1  1  microprocessors  executing  them,  but  it  is 
lkr»|y  that  similar  tasks  in  real  applications  would  be  shorter,  even  on  LSI-1  1's 
Another  aspect  of  the  slowness  of  t. SI-1  1's  is  that  interprocessor  communication 
times  (the  parameter  r  in  Section  Mil)  are  relatively  long  on  the  MuNet  To  bung 
the  ratio  of  communication  to  compute  times  closer  to  what  one  can  realistically 
expect  in  the  future,  the  compute  time  w  is  artificially  lengthened  (it  is  true  that 
this  also  makes  the  prrforniance  data  look  more  pleasing)  The  only  case  where 
this  was  not  done  was  in  the  eight  queens  program 

f  igure  7  1  gives  an  example  of  the  direct  output  from  the  performance  tests 
As  can  be  seen,  the  time  per  event  (which  is  the  reciproca'  of  the  throughput) 
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I  fatso  graphs  show  raw  performance  c lata  obtained  from  tfat  MuNet  on  both 
tfa>  asynchronous  test  (fop  graph)  and  fho  synchronous  tost  (bottom  graph) 

I  fat  “degree  of  parallelism"  is  tho  numfa'r  of  events  simultaneously  active  in 
tfat  systom  7 fat  "time  per  event."  or  inverse  thr oughput  is  givon  both  in  the 
aggregate  (top  curve  m  <>,*  h  graph)  arte!  t>y  <  om/Kitt-nls  ( labeled  by  name  in 
tfat  top  graph).  7 fa>s>  components  indicate  the  fr.f  tions  of  CPU  tirm< 
devoted  to  different  activities  on  the  metering  processor  only  7 fa-  graphs 
show  that,  on  this  three- processor  topology .  throughput  increases  by  a  factor 
of  between  fwo  and  th  ee  /n  the  presence  of  enough  parallelism 


figure  7  1  MuNet  performance  ilati 


starts  out  at  a  high  level,  when  the  degree  of  pnra'lensm  is  too  low  to  cause  any 
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diffusion  to  take  place,  then  drops  and  levels  off  at  a  new,  lower  value  as  the 
QFUDGl  threshold  is  crossed  tor  the  experiments  that  were  conducted,  this 
asymptotic  value  of  the  time  per  event  did  not  seem  much  affected  by  the  exact 
QfUDGF  used,  only  the  precipitousness  of  the  drop  toward  that  asymptote 
changed  thus  It  is  reasonable  to  associate  this  asymptotic  value  with  the 
throughput  or  the  network,  when  sufficient  events  are  present,  and  compare  This 
with  theoretical  predictions  derived  from  the  linear  programming  model  this  is  done 
In  I  igure  7  ? 

the  reasonable  agreement  of  actual  and  theoretic  at  results  in  figure  7  ?(a) 
Indicates  that  trie  diffusion  strategy  is  doing  about  ns  good  a  job  as  can  be  done  in 
this  case  not  a  particularly  surprising  result  the  lack  of  greater  improvement  in 
Actual  performance  between  the*  one  and  t wo-processor  cases  Is  a  result  of  ’tie 
performance  guirk  discussed  earlier,  winch  occurs  as  the  original  processor  gets  its 
first  neighbor 

The  actual  experience  with  parallel  topologies  shown  In  Figure  7  ?(b)  eventu¬ 
ally  diverges  from  the  theoretical  because  of  a  serious  case  of  tins  performance 
quirk  These  curves  cannot  really  be  seen  in  a  proper  liqht  except  by  comparison 
with  the  curve  labe'ed  "QFUDGF  *  which  shows  the  performance  characteristics 
of  the  topology  when  all  events  are  constrained  to  execute  on  the  metering  proc¬ 
essor  As  the  number  of  processors  passes  four  (the  metering  processor  plus 
three  neighbors),  throughput  drops  almost  to  zero  F  xamination  of  the  detailed 
curves  U.t  this  ri.se  reveals  that  the  mete  ing  processor  is  spending  virtually  all  its 
time  in  communication  overhead  Tins  is  because  MuNet  processors  give  communi¬ 
cation  functions  un  onditional  priority  over  event  execution,  with  three  or  four 
neighbors,  at  least  one  of  them  is  almost  certain  to  have  something  to  send  a  proc¬ 
ess!  r  a  any  .in  en  ''mi  Thus  tf'ere  is  very  little  chance  for  events  to  execute  on 
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time  per  event 


number  of  processors  — - ^ 


(a)  Stack  topology  (as  shown  in  Figure  6  7) 


12  3  4  6 

number  of  processors  ^ 


(b)  Parallel  topology  (as  shown  in  Figure  R  0) 

Parameters  used  for  theoretical  modef  f  *  0  R  seconds,  c  -  30  ms 

T hose  graphs  show  (he  <r  tua/  throughput  chart*  ter i sties  of  (wo  Kurds  of 
MuNot  topologies  in  the  presence  of  large  numbers  of  concurrently  execut¬ 
ing  events,  for  both  th>>  asyrn  hronous  and  syn<  hronous  tests.  7 hecrrr't ical 
predict  ions  derived  from  the  linear  programming  model  of  Chapter  6  are 
Included  for  comparison.  Anomalies  in  the  lower  graph  are  discussed  in  the 
text. 


Figure  7  ?  Throughput  of  the  MuNot 


the  mi  termg  processor  When  other  processors  are  busy  executing  events,  this 
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oftect  is  ameliorated  somewhat,  but  it  still  eventually  makes  itself  felt 

finally,  figure  /.3  stiows  the  results  of  executing  the  eight  queens  program 
the  shape  of  this  curve  compares  favorably  with  those  of  figure  / .i*(b).  a  reason 
for  optimism  that  the  basic  diffusion  strategy  can  in  fact  be  useful  over  some  tea 
sonable  range  of  applications 


100 

sec 


A 


60 

sec 

time  for  solution 


1  2  3  A  6 


number  of  processors  ^ 

topology  -  parallel  topology  (as  shown  In  figure  0  6) 
QfUOGI  =  10 


This  graph  shows  fhe  armx/nf  of  reK,l  time  'equ.rnd  by  the  MuNet  to  solve  /he 
eight  queens  problem,  using  different  numbers  of  processors. 

figure  /.3  fight  queens  performance  results 
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7.1.4:  Diffusion  of  Object* 

the  foregoing  discussion  has  concentrated  largely  on  the  diffusion  of  events 
through  the  network  to  take  greater  advantage  of  the  computing  power  available 
It  is  equally  reasonable  to  think  of  diffusion  as  a  distribution  strategy  tor  objects  to 
take  better  advantage  of  the  memory  space  collectively  available  among  the  nodes 
In  a  network  Of  course,  even  without  explicit  diffusion  of  objects,  diffusion  of 
events  does  imply  a  certain  strategy  for  distributing  objects  about  a  network 
Once  an  event  has  diffused  to  some  location  where  it  is  to  be  executed,  objects  it 
accesses  must  either  already  be  present  there  or  be  brought  there:  otherwise, 
ttie  location  of  the  event  will  not  have  been  determined  solely  by  diffusion,  but  also 
by  othor  factors  this  latter  may  well  be  c  preferable  state  of  affairs,  but 
represents  a  departure  from  the  pure  event  diffusion  strategy 

The  moral  of  the  story  is  that  the  distribution  strategy  for  objects  cannot  be 
divorced  from  th'*  for  events,  and  if  independent  diffusion  strategies  are  to  be 
used  for  each,  then  some  policy  must  be  adopted  for  resolving  conflicts  when  they 
arise  A  reasonable  approach  would  be  to  use  object  diffusion  to  secrete  away 
little-used  objects  wherever  there  is  extra  memory  space,  but  *et  frequently 

accessed  objects  be  distributed  so  as  to  be  near  the  events  that  use  them 

The  object  distribution  question  has  other  facets  as  well  When  and  where 

should  multiple  copies  of  an  object  be  made7  What  kind  of  strategy  will  keep 

related  objects  on  the  same  or  neighboring  processors,  to  minimize  reference  tree 
overhead  and  make  sure  that  once  an  event  has  been  moved  to  a  processor  where 
there  is  a  copy  of  one  object  it  needs,  copies  of  other  useful  objects  will  be  avail¬ 
able  nearby7  In  general,  it  will  probably  be  harder  to  operate  effk'.iently  in  an 
environment  where  objects  must  be  distributed  due  to  lack  of  memory  space  than 
one  in  which  events  must  be  distributed  duo  to  lack  of  computing  power. 
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Nevertheless,  the  questions  cannot  be  completely  separated,  so  the  next  section 
makes  a  stab  at  a  better  strategy  for  event  and  object  distribution 

7.2:  Extensions  to  the  Diffusion  Strategy 
7.2.1:  Pull  Factors 

The  main  problem  with  the  simple  diffusion  strategy  is  not  its  concept  of 
spreading  the  load  across  the  network,  but  Its  naive  assumption  that  events  (or 
objects)  are  all  interchangeable,  and  that  sending  one  event  out  in  a  particular 
direction  is  as  good  as  sending  another  In  fact,  different  events  may  well  have 
"roots"  (l.o. ,  copies  of  objects  thoy  access)  in  diffeient  directions,  and  if  it  is 
desired  to  send  an  event  in  a  particular  direction  for  load  balancing,  some  events 
may  be  much  more  appropriate  for  this  purpose  than  others 

To  reflect  this  distinction,  one  might  imagine  adding  "pull  factors"  to  the  basic 
diffusion  strategy  If  an  event  were  being  moved  closer  to  copies  of  the  texts  of 
any  objects  it  referenced,  the  QfUDGE  in  effect  for  moving  the  event  in  that  direc¬ 
tion  would  be  reduced  by  tho  number  of  such  objects  times  the  pull  factor  If  an 
event  were  being  moved  away  from  copies  of  the  texts  of  objects  it  referenced, 
multiples  of  the  pull  factor  would  be  added  to  QfUDGE.  The  event  would  thus 
experience  a  "pull"  in  whatever  direction  most  of  the  object  texts  likely  to  be  use¬ 
ful  to  It  might  lie  This  strategy  is  analogous  (assuming  for  the  moment  that  the 
distribution  of  objects  has  been  fixed  In  advance)  to  attaching  a  spring  between 
each  event  and  every  object  it  references,  then  allowing  the  events  to  be  pulled 
around  un»il  equilibrium  is  reached  A  form  of  our  original  diffusion  strategy  can  be 
incorporated  into  this  analogy  by  imagining  a  repulsive  force  between  any  pair  of 
events,  encouraging  them,  other  things  being  equal,  to  spread  as  widely  as  possible 
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across  the  network 


Just  as  pull  factors  can  be  used  to  improve  the  diffusion  strategy  for  events, 
they  can  be  used  with  the  diffusion  strategy  tor  objects  Of  course,  the  use  of 
pull  factors  alone  would  tend  to  pull  all  objects  together  onto  one  jirocessor  Such 
a  distribution  might  well  demand  memory  space  In  excess  of  that  available  on  that 
j>rocessor,  and  does  not  open  any  vory  interesting  possibilities  tor  the  distribution 
of  the  events  that  use  those  objects  those  problems  can  be  solved  by  counter¬ 
balancing  the  attractive  force  of  object  references  with  a  repulsive  force  tending 
to  move  objects  away  from  high  concentrations  of  other  objects  the  synergy  of 
those  two  forces  would  then  cause  tightly  related  groups  of  objects  to  cluster 
together  on  the  same  processor,  while  objects  more  tenuously  related  would  be 
stored  farther  away 

To  get  some  idea  of  the  possible  effects  of  this  strategy,  some  simulations 
were  carried  out  using  a  structure  of  Interreferencing  objects  that  would  be 
characteristic  of  numerical  Iterations  over  two-dimensional  arrays  of  grid  points,  as 
in  solutions  of  1  aplace's  equation  or  other  sets  of  partial  differential  equations 
Semantically,  one  may  imagine  those  objects  arranged  in  a  two-dimensional  grid 
fthis  says  nothing  about  their  actual  physical  distribution  in  a  network)  such  that 
each  object  has  references  to  its  neighbors  on  the  north,  east,  west,  and  south 
Objects  n  ong  the  edges  of  ttie  grid,  c  f  course,  have  fewer  neighbors 

The  iterative  computation  to  bo  performed  over  these  grid  point  ohjocts  can  be 
described  in  general  terms  as  follows  When  an  iteration  is  to  begin,  this  tact  Is 
signalled  to  each  grid  point  f  ach  grid  point  then  sends  to  each  of  Its  neighbors 
the  current  values  of  the  relevant  state  var.ahles  at  that  point.  When  a  grid  point 
object  has  received  these  state  varlahle  values  from  all  its  neighbors,  those  values 
are  combined  with  ttie  .  erronf  state  at  the  grid  point  to  produce  the  now  state 
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that  will  reign  at  that  grid  point  when  the  iteration  is  complete 


As  our  running  example  of  object  distribution  strategies,  we  will  assume  a 
1 2x  1  2  array  of  grid  point  objects,  mapped  onto  a  4x4  square  network  of  proces¬ 
sors  Since  each  iteration  of  the  computation  results  in  a  side  effect  (updating  the 
state  variables)  to  each  grid  point  object,  there  is  little  point  In  storing  multiple 
copies  of  any  of  these  objects  We  will  assume  that  the  object  distribution  algo¬ 
rithm  is  aware  of  this  and  keeps  only  a  single  copy  of  the  text  of  each  object 
Thus  it  is  meaningful  to  talk  about  the  location  of  a  particular  object  text 
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Figure  7.4  Ideal  distribution  of  grid  point  objects 


The  ',obvlous,■  ideal  assignment  of  objects  to  processors  in  this  example  is  the 
asslgnmont  of  a  3x3  subarray  of  grid  point  objects  to  each  processor,  such  that 
adjacent  subarrays  are  assiqned  to  adjacent  processors.  This  assignment  is 
depicted  in  Figure  7  4  The  12x1?  array  of  squares  in  this  figure  represents  the 
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12x12  array  of  grid  point  objects  Within  the  square  corresponding  to  each  object 
Is  shown  a  picture  of  the  reference  tree  for  that  object  I  he  4x4  array  of  dots  in 
each  picture  represents  the  4x4  processor  array,  the  processor  marked  with  an 
"x"  is  the  processor  where  the  object's  text  is  stored  Solid  lines  mark  the 
Interprocessor  links  that  comprise  the  object's  reference  tree  the  reader  can 
verify  that  in  every  case  this  tree  includes  every  processor  containing  o  reference 
to  the  object 

Results  of  ajiplying  the  simple  diffusion  strategy  to  distributing  objects  are 
shown  in  the  series  of  snapshots  in  I  igure  7  b  figure  7.5(a)  shows  the  initial 
configuration,  with  all  grid  points  stored  on  the  lower  left  hand  processor  To  obtain 
each  subsequent  snapshot  in  I  igure  I  5  from  the  previous  one.  one  pass  was  made 
over  the  entire  set  of  grid  point  objects  (In  a  randomly  determined  order),  and  the 
simjilo  diffusion  criterion  applied  to  each  one  to  see  if  it  should  be  moved  to  a 
neighboring  processor  This  is  not  an  exact  simulation  of  the  way  in  which  things 
would  happen  on  a  real  reference  tree  network  (presumably  there  would  be  more 
concurrency),  but  the  results  should  he  fairly  similar  In  this  case,  we  can  see  that 
the  result  is  a  series  of  long,  strotrhed-out  reference  trees 
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Figure  7  5(b)  After  one  pass  using  the  simple  diffusion  strategy 
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Figure  7. 6(e):  After  six  passes  using  the  simple  diffusion  strategy 
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Results  of  using  a  pull  factor  along  with  the  simple  diffusion  strategy  are  shown 
in  F  igure  /'.(>  Although  many  reference  trees  still  become  larger  than  in  the  ideal 
situation,  there  are  certainly  several  "domains"  of  neighboring  objects  that  have  all 
been  assigned  to  the  same  processor,  and  the  growth  of  reference  trees,  although 
still  far  greater  than  In  the  ideal,  is  not  nearly  as  luxuriant  as  in  Figure  7  6 
Another  thing  to  notice  about  Figure  7. 6  is  that  the  use  of  pull  factors  has  put 
enough  of  a  damper  on  diffusion  that  not  all  processors  are  being  used  equally  In 
fact,  several  processors  on  the  upper  right  have  not  been  used  at  all  Thus  lower 
communication  costs  have  been  achieved  at  the  expense  of  an  increased  computa¬ 
tional  load  at  some  processors  Depending  on  the  ratio  of  these  costs,  this  may  or 
may  not  represent  a  good  compromise. 
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Figure  7  6(a)  After  one  pass  using  pull  factors 
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7.2.2:  Variable  Pull  Factors 


Pull  factors  manage  a  significant  improvement  over  the  simple  diffusion  strat¬ 
egy.  but  t/iere  is  still  clearly  a  great  deal  of  room  for  improvement  Unfortunately, 
we  are  coming  to  the  edge  of  information  that  a  scheduling  algorithm  might  reason¬ 
ably  be  expected  to  be  able  to  discover  for  itself  In  order  to  make  much  further 
progress,  some  scheduling  information  supplied  either  by  the  user  or  by  some 
language  processor  will  have  to  be  included  with  objects  and  events  In  general,  it 
is  desirable  for  this  information  to  be  expressed  in  as  topology-independent  a  form 
as  possible,  to  make  it  applicable  to  running  the  same  program  on  a  wide  variety  of 
different  reference  tree  networks  Even  within  this  constraint,  the  most  appropri¬ 
ate  form  for  scheduling  specifications  will  vary  for  the  array-processing  example 
dealt  with  in  the  previous  section,  the  best  form  of  specification  might  be  an  indica¬ 
tion  that  the  grid  point  objects  fit  on  a  Cartesian  grid,  giving  the  co-ordinates  of 
each  object  on  the  grid  This  section  presents  a  simpler  scheme  that  should  be 
suitable  for  many  less  structured  tasks 

One  problem  with  pull  factors  is  that  every  reference  contained  in  an  event  or 
object  text  exerts  an  equally  strong  pull  For  the  array-processing  example  given 
above,  this  is  not  especially  a  liability,  but  in  most  cases  some  objects  referenced 
are  much  more  likely  to  actually  bo  accessed  than  others.  For  example,  an  event 
should  be  pulled  hardest  by  its  target  object  Other  object  references  in  the 
event  might  be  passed  along  for  several  generations  without  the  corresponding 
objects  ever  being  accessed;  to  the  extent  that  this  is  true,  such  references 
should  exert  less  pull  Similar  remarks  can  be  made  concerning  pulls  on  objects 
Certain  references  contained  in  a  text  may  be  much  more  likely  than  others  to  be 
followed  by  anyone  examining  the  text,  the  references  most  likely  to  be  used 
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should  exert  the  strongest  pull 

It  different  references  contained  in  an  event  or  object  text  are  to  exert 
different  degrees  of  pull  on  it,  the  question  arises  of  whether  the  strengths  of  pull 
should  be  specified  in  the  event  or  object  text  being  pulled,  a  particular  strength 
being  assigned  to  each  slot  containing  a  reference,  or  whether  the  strength  of  pull 
should  be  determined  by  the  object  being  referenced,  so  that  a  given  object  will 
pull  with  the  same  force  on  every  event  or  object  text  referencing  it  In  other 
words,  should  the  strength  of  pull  be  a  property  of  the  referencer  or  of  the 
referencee?  A  strong  argument  for  it  being  a  property  of  the  referencer  Is  that  an 
object  is  not  equally  likely  to  be  accessed  over  all  paths  by  which  it  might  be 
reached  f  or  example,  an  objoct  named  as  the  target  object  of  an  event  is  cer¬ 
tain  to  be  accessed  by  the  execution  of  that  event  A  reference  to  the  same 
object,  entered  In  a  directory  of  a  file  system,  might  be  passed  over  almost  every 
time  the  directory  was  examined  Even  on  the  occasions  when  the  reference  was 
retrieved  from  the  directory,  another  long  interval  might  elapse  without  the  associ¬ 
ated  object  text  actually  being  accessed  It  stands  to  reason  that  the  reference 
to  this  objoct  should  exert  a  much  stronger  pull  on  the  event  than  on  the  directory 

7.3:  Communication  Graphs 

Thus  far,  the  scheduling  strategies  suggested  all  have  a  fairly  mechanistic 
flavor  a  user  with  a  program  to  run  might  have  the  operation  of  the  scheduling 
mechanism  explained  to  him  and  be  introduced  to  the  changeable  parameters 
accessible  to  tvm,  but  then  he  would  be  on  his  own  There  would  be  no  guidance 
other  than  experiment  to  tell  the  user  which  parameter  values  would  best  accom¬ 
plish  what  he  wants,  namely  to  run  his  program  as  quickly  as  possible  The  user  Is 
being  asked  to  inform  the  system  about  the  nature  of  his  program,  so  that  the 
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system  can  execute  it  efficiently  But  the  form  of  the  specification  Is  so  alien  to 
the  way  in  which  the  user  might  describe  the  program  that  the  only  way  of  telling 
which  of  two  specifications  is  ttie  more  accurate  is  to  experiment  with  them  and 
see  which  results  in  faster  execution1 

The  situation  faced  by  a  user  of  this  mechanism  Is  similar  to  that  confronted  by 
a  designer  of  real-time  systems  whose  only  means  for  influencing  system  behavior 
is  by  assigning  priorities  to  tasks  What  the  user  would  like  to  do  is  specify  the 
required  real-time  behavior,  which  could  then  bo  satisfied  by  the  system  in  any  suit¬ 
able  manner  What  he  ends  up  doing  is  tinkering  with  priorities  of  tasks  until  the 
system  appears  to  meet  the  required  real-time  demands  This  is  bad  because 
there  is  no  formal  framework  to  indicate  what  sets  of  priorities  should  be  used,  and 
there  is  often  no  assurance  that  the  sot  finally  chosen  will  really  guarantee  the 
required  performance  under  all  circumstances 

Similarly,  it  would  be  better  if  a  user  of  a  reference  tree  network  could  simply 
describe  ttie  behavior  of  his  program  in  reasonably  high-level  terms,  rather  than 
indulge  in  tweaking  little-understood  fudge  factors  One  possibility  along  these 
lines  would  be  for  the  user  (or  some  programming  language  processor)  to  supply  a 
"communication  graph"  showing  the  frequency  of  communication  between  pairs  of 
modules  in  the  program 

The  actual  way  in  which  communication  would  happen  between  a  pair  of  VIM 
objects  is  that  an  event  or  a  set  of  related  events  would  access  both  objects  If 
copies  of  the  texts  of  the  objects  were  stored  at  some  distance  from  each  other, 
then  either  object  texts  or  events  or  both  would  have  to  be  moved,  incurring  com¬ 
munication  costs  roughly  proportional  to  the  original  distance  between  the  object 
texts.  A  communication  graph  would  have  a  node  for  each  object,  and  an  arc  for 
each  pair  of  objects  that  might  communicate.  Each  arc  would  be  labeled  with  the 


198 


Chapter  7.  Scheduling  Strategies  for  Reference  Tree  Networks 


communication  cost  incurred  per  unit  distance  that  the  corresponding  pair  of 
objects  were  separated  The  object  distribution  problem  would  then  reduce  to  the 
problem  of  finding  an  assignment  of  objects  to  processors  that  minimized  communi¬ 
cation  costs 

the  obvious  assignment  that  minimizes  total  communication  costs  is  the  assign¬ 
ment  of  all  objects  to  the  same  processor  In  some  cases,  this  might  be  ruled  out 
by  memory  constraints  In  most  other  cases,  this  would  be  undesirable  because  it 
would  encourage  all  events  using  the  objects  to  congregate  on  the  same  proces¬ 
sor,  resulting  in  poor  load  sharing  Perhaps  some  arcs  should  have  negative  commu¬ 
nication  costs,  indicating  that  the  objects  involved  do  not  communicate,  and  further¬ 
more  that  If  they  are  assigned  to  the  same  processor,  events  using  them  will  be 
competing  for  CPU  time  It  must  be  said,  however,  that  this  solution  is  rather  ad 
hoc.  and  that  the  problem  needs  more  study. 

Another  aspect  of  object  distribution  strategy  is  the  decision  as  to  whether  to 
make  multiple  copies  of  an  object  The  wisdom  of  this  will  depend  on  the  relative 
frequencies  of  accesses  and  side  effects  to  the  object.  A  good  way  of  conveying 
this  information  in  a  communication  graph  might  be  to  label  each  node  with  the  cost 
of  dispersing  copies  of  the  corresponding  object  over  some  unit  distance  This 
cost  would  be  high  if  side  effects  to  tho  object  wore  expected  to  be  frequent; 
otherwise  it  would  be  lower 

Making  the  best  use  of  all  the  information  in  a  communication  graph  is  almost 
certainly  an  NP-complete  problem,  however,  It  Is  an  open  question  what  kinds  of 
approximate  methods  might  be  able  to  produce  good  enough  results  to  make  the 
approach  worthwhile  Other  open  questions  exist  as  well  Can  a  communication 
graph  be  useful  to  a  distributed  scheduler  each  of  whose  components  operates 
strictly  on  the  basis  of  local  information,  or  must  global  knowledge  of  the  network 
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topology  be  available?  How  can  the  communication  graph  concept  deal  with  a 
dynamic  situation  in  which  objects,  and  hence  potentially  nodes  and  arcs,  are  being 
created  and  garbage-collected?  How  is  all  the  detailed  information  in  a  communica¬ 


tion  graph  to  be  supplied'7  Clearly  communication  graphs  are  currently  a  long  way 
from  being  a  practical  medium  for  scheduling  specifications,  but  the  development  of 
some  such  mechanism  for  allowing  the  nature  of  a  program  to  be  communicated 
more  straightforwardly  is  an  important  goal  for  the  future. 

7.4:  Summary 

This  chapter,  more  than  most  in  this  thesis,  raises  many  more  questions  than  it 
answers  A  simple  scheduling  strategy,  based  on  "diffusion"  of  events  and  objects 
to  less-loaded  processors,  seems  to  do  a  reasonable  job  of  load  balancing,  but 
does  not  obtain  any  guidance  from  the  relationships  among  data  items  and  may 
therefore  increase  communication  costs  disastrously.  An  example  of  such  a  disas¬ 
trous  Increase  is  given  in  Figure  7.5.  Incorporation  of  fixed  or  variable  "pull  fac¬ 
tors"  ameliorates  this  situation  considerably  (see.  e.g..  Figure  7.6),  but  still  leaves 
an  undesirable  situation  in  several  respects. 

One  attribute  of  the  "pull  factors"  solution  which  might  be  regarded  as  a 
feature  is  that  it  provides  a  large  number  of  parameters  through  which  the  user 
can  twiddle  the  performance  of  the  system  Unfortunately,  the  relationship 
between  the  structure  of  a  program  and  the  optimum  values  of  these  parameters  is 
often  obscure  The  concept  of  communication  graphs  is  an  attempt  to  bring  the 
level  of  scheduling  specifications  closer  to  the  level  at  which  the  user  thinks  about 
his  program,  but  many  details  of  both  the  structu'o  and  use  of  such  graphs  remain 
to  be  worked  out.  Whether  communication  graphs  or  pull  factors  are  used,  how¬ 
ever,  there  is  still  a  large  amount  of  scheduling  information  to  supply.  How  much  of 
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that  can  be  calculated  by  some  language  processor  scanning  the  user's  program, 
and  how  much  must  be  supplied  by  the  user  himself? 

Lven  though  the  Introduction  of  pull  factors  works  a  marked  improvement  In  the 
performance  of  the  diffusion  strategy,  many  situations  are  still  treated  in  far  from 
the  ideal  manner  (compare  Figures  7  6  and  7. 4)  It  Is  hard  to  know  what  improve¬ 
ments  over  the  pull  factor  method  can  bo  achieved  by  using  communication  graphs 
or  the  like,  since  the  performance  achieved  using  these  depends  on  just  how  the 
information  contained  in  them  is  analyzed  and  used  by  the  network.  It  is  reason¬ 
able  to  conjecture,  though,  that  no  such  method  will  match  the  results  of  doing  the 
distribution  by  hand,  as  in  Figure  7  4 

To  keep  from  being  too  pessimistic  about  this  state  of  affairs,  it  is  worthwhile 
to  make  an  analogy  with  paging  strategies  on  virtual  memory  systems  These  strat¬ 
egies,  too,  must  make  object  distribution  decisions  (between  primary  and  secondary 
memory)  and  respond  to  changing  conditions  These  strategies,  too,  attempt  to 
make  decisions  based  on  the  observed  behavior  of  'unnmg  programs,  and  these 
strategies,  too,  could  benefit  from  being  told  more  about  the  exact  nature  of  a  pro¬ 
gram  being  executed  Yet  in  most  cases  programmers  have  refrained  from  fiddling 
with  the  virtual  memory  strategy  or  taking  matters  out  of  the  paging  system's 
hands  by  doing  their  own  input/output  explicitly  There  are  several  reasons  for 
this.  One  is  that  for  many  large  systems  of  programs,  the  optimal  scheduling  strat¬ 
egy  is  no  less  obvious  to  the  system  than  to  the  user  Another  is  that  paging 
strategies  do  well  enough  that  it  is  ordinarily  not  worthwhile  for  the  programmer  to 
try  to  squeeze  that  last  ounce  of  performance  from  the  system  With  hardware 
costs  decreasing,  this  argument  becomes  more  and  more  compelling  Although  there 
are  several  ways  in  which  the  virtual  memory  problem  is  less  complicated  than  the 
reference  tree  network  scheduling  problem,  a  reasonable  research  goal  is  to  make 
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the  two  reasons  given  above  valid  statements  about  reference  tree  networks  as 
well  Then  users  can  go  back  to  getting  their  algorithms  right,  rather  than  fussing 
with  tunable  system  parameters. 

The  performance  results  presented  in  this  chapter  are  very  preliminary,  but  at 
least  they  are  not  discouraging.  Much  work  with  more  substantial  applications 
remains  to  be  done  to  determine  the  adequacy,  or  inadequacy,  of  even  the  current 
scheduling  strategies  Beyond  this,  it  must  be  seen  if  reasonable  scheduling  strat¬ 
egies  can  be  built  around  the  cherished  reference  tree  network  philosophy  of 
decision-making  on  the  basis  of  local  information  only,  or  whether  more  global  infor¬ 
mation  and  interaction  is  required.  Although  it  would  be  nice  to  have  a  beautiful 
theory  to  answer  these  questions,  it  seems  that  a  great  deal  of  experimentation 
lies  ahead 
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The  future  development  of  computer  systems  will  be  influenced  by  two  thrusts 
of  technological  progress  the  ability  to  build  circuits  which  are  faster,  and  the 
ability  to  build  circuits  which  are  bigger  (/.e..  which  have  greater  numbers  of  logical 
components).  The  former  thrust  will  lead  to  continued  improvement  in  what  is 
attainable  using  architectures  that  are  now  in  vogue,  but  the  latter  thrust  poses  a 
challenge  to  develop  whole  new  architectures  If  the  construction  of  a  computing 
system  with  a  greater  number  of  components  is  to  render  it  more  powerful,  there  is 
no  choice  but  to  design  a  system  in  which  more  and  more  operations  can  be  per¬ 
formed  concurrently.  In  an  architecture  based  on  strict  sequentiality,  once  the 
logic  necessary  for  all  functions  has  been  provided,  it  is  difficult  to  achiove  any 
further  speed  Increase  by  adding  more  logic,  since  what  matters  Is  the  total 
number  of  operations  needed  to  implement  a  function,  not  the  variety  of  physical 
gates  which  perform  those  operations. 

The  technological  opportunity  to  build  more  complex  circuits,  if  they  can  be  put 
to  use,  can  be  exploited  to  some  extent  by  switching  to  architectures  which  are 
internally  parallel,  even  though  they  still  give  a  superficial  appearance  of  sequen¬ 
tiality.  However,  to  take  full  advantage  of  the  fruits  of  the  very-large-scale 
Integration  (VLSI)  revolution,  the  user  must  be  introduced  to  the  world  of  con¬ 
currency  and  encouraged  to  express  his  algorithms  in  as  parallel  a  fashion  as  pos¬ 
sible.  The  purpose  of  the  research  described  in  this  thesis  has  been  to  develop 
methodologies  for  accomplishing  this 

More  specifically,  this  research  may  be  described  as  a  search  for  ways  of 
making  multiprocessor  systems  usable.  The  word  "multiprocessor”  is  intended  to 
convey,  first  of  all,  the  presence  of  concurrency,  and  secondly,  the  idea  that 
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general-purpose  processing  is  going  on.  This  Is  not  meant  to  imply  any  restriction 
on  the  physical  incarnation  of  that  processing  power,  but  is  Intended  to  narrow  our 
scope  to  include  only  systems  capable  of  general-purpose  computation.  The  archi¬ 
tecture  of  special-purpose  VLSI  systems,  for  example,  although  an  important  topic 
in  its  own  right,  is  not  a  topic  addressed  by  this  thesis 

Within  the  sphere  of  general-purpose  multiprocessor  systems,  the  strategy  of 
this  thesis  has  been  to  make  a  broad  cut  across  the  entire  domain,  covering  both  a 
proposed  user  interface  to  such  systems,  an  implementation  supporting  that  Inter¬ 
face.  and  a  preliminary  evaluation  of  that  implementation.  It  is  appropriate  to 
review  the  progress  made  in  each  of  these  areas 

ft.  1 :  The  VIM  Virtual  Machine 

The  VIM  virtual  machine  is  only  one  of  many  that  have  been  suggested  for 
parallel  computation:  communicating  sequential  processos[  1 9]  and  data  flow[7,30] 
are  notable  examples  of  alternatives  On  the  more  concrete  side,  the  virtual 
machines  presented  by  modern  timesharing  systems,  such  e*  MlllTICS[31]  or 
UNIX[29],  deal  with  interaction  between  concurrent  activities  These  systems, 
however,  frequently  either  fail  to  provide  facilities  powerful  enough  to  support  all 
the  kinds  of  interaction  that  might  be  desirable  (the  limitations  of  UNIX  pipes  are  an 
example  of  this),  or  provide  facilities  difficult  to  support  on  many  multiprocessor 
systems  (e.g.,  memory  shared  between  processes).  A  final  virtual  machine  for 
parallel  computation,  the  actor  machine  of  PLASMA[18],  Is  the  closest  cousin  to 
VIM.  The  design  of  VIM  was  heavily  influenced  by  the  actor  model  of  computation, 
and  in  fact  VIM  may  be  thought  of  as  a  reduction  to  practice  of  many  of  the  con¬ 
cepts  embodied  in  the  actor  model. 

The  interrelationships  of  the  various  parts  of  VIM  have  already  been  discussed 
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in  Chapter  2.  particularly  in  Section  27,  and  will  not  be  further  dealt  with  here  By 
way  of  comparison  with  other  alternatives,  however,  It  is  worth  recapitulating  the 
advantages  of  the  VIM  approach  First  of  all.  VIM  is  simple  and  tractable  It  can 
be  described  precisely  by  the  blackboard  interpreter  of  Section  2.5;  that  inter¬ 
preter  could  be  modified  easily  to  cover  the  enhancements  proposed  in  Chapter  3 
The  essential  simplicity  of  VIM  stems  from  Its  roots  in  the  mu  calculus[  1 5,38],  a 
simple  syntactic  formalism  for  message-passing  computation 

Second,  VIM  is  flexible  and  adaptable  This  flexibility  manifests  itself  in  two 
ways  In  the  ability  to  create  complex  structures  such  as  sophisticated  synchroni¬ 
zation  operators  out  of  the  basic  set  of  VIM  primitives,  and  in  VIM's  amenability  to 
extension  to  support  additional  demands  imposed  by  desires  to  implement  various 
operating  system  functions,  as  illustrated  in  Chapter  3 

Third,  VIM  is  designed  for  efficient  execution  on  hardware  of  a  familiar  nature 
Although  it  does  support  actor-style  message  passing  and  a  garbage-collected 
space  of  objects,  two  concepts  that  have  not  often  been  associated  with  run-time 
efficiency,  VIM  also  allows  direct  access  to  object  texts  and  permits  the  composi¬ 
tion  of  several  functions  into  the  text  of  one  type  object,  obviating  some 
message-passing  activity  at  the  very  lowest  level.  As  time  passes  and  hardware 
designs  progress,  this  attribute  of  VIM  may  well  decrease  in  importance 

Finally  and  most  importantly.  VIM  is  a  powerful  and  effective  tool  Its  orienta¬ 
tion  toward  objects  corresponds  with  various  modern  views  about  how  to  structure 
programming  systems[5,2iJ  J.  and  its  ability  to  handle  parallelism  suits  it  ideally  to 
multiprocessor  systems  The  various  guarantees  that  VIM  makes  with  respect  to 
the  co-ordination  of  event  executions  and  object  accesses  reduce  the  range  of 
possible  execution  histories  a  programmer  must  consider  and  thereby  simplify  his 
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VIM  is  hardly  a  finished  product,  though  The  operating  system  support  exten¬ 
sions  discussed  in  Chapter  3  need  to  be  worked  out,  implemented,  and  reconciled 
with  the  ideas  set  forth  by  Gula[13,14j  on  structuring  a  MuNet  operating  system 

Significantly,  "machine-level"  VIM  is  not  a  very  easy  language  to  program  In, 
despite  the  various  simplifying  guarantees  made  by  it  This  is  primarily  because 
many  of  the  machine-level  concepts  are  at  too  low  a  level  for  everyday  use  The 
need  to  avoid  resource  requests  after  performing  a  side  effect  during  the  same 
event  execution,  for  example.  Is  a  constant  distraction  Additionally,  many  useful 
concepts,  such  as  procedures  and  environments,  must  be  synthesized  out  of  more 
primitive  facilities  available  at  the  machine  level  Thus  some  higher-level  program¬ 
ming  language,  such  as  MuSpeak[34,35],  is  vitally  needed  to  make  practical  use  of 
VIM  An  interesting  question,  in  fact,  is  the  extent  to  which  the  nature  of  the 
basic  VIM  execution  environment  should  be  visible  to  a  user  of  such  a  language  It 
cannot  be  argued  that  the  simplifying  guarantees  made  by  VIM  actually  uncompli¬ 
cate  the  programmer's  task  unless  it  can  be  shown  how  those  guarantees  affect 
the  environment  in  which  the  programmer  works  If  all  characteristics  of  VIM  are 
hidden  snugly  beneath  a  rather  different-appearing  programming  language,  then  the 
use  of  VIM  may  be  a  convenience  to  the  language  implementor,  but  can  be  of  no 
concern  to  the  user. 

Use  of  the  VIM  virtual  machine  can  help  make  a  multiprocessor  system  simple, 
flexible,  efficient,  and  powerful.  This  is  true  whether  or  not  any  of  the  other  imple¬ 
mentation  mechanisms  suggested  in  this  thesis  are  employed  However,  the  design 
of  languages  to  serve  as  the  primary  interface  between  programmers  and  such  a 
system  remains  an  open  (and  important)  research  question  The  extent  to  which 
the  characteristics  of  VIM  are  visible  through  this  interface  will  help  determine  the 
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real  significance  of  the  VIM  concept  as  a  good  organization  for  multiprocessor  sys 


terns. 

8.2:  Reference  Trees 

Reference  trees  are  the  principal  implementation  mechanism  proposed  in  this 
thesis.  Although  the  VIM  virtual  machine  could  be  used  on  any  kind  of  computer 
system,  reference  trees  only  make  sense  on  certain  architectures,  notably  those 
discussed  in  Section  4  1  Then  aga.n,  on  these  architectures,  reference  trees 
could  be  used  to  support  an  environment  quite  different  from  VIM 

fhe  principal  attractions  of  the  reference  tree  mechanism  are  its  simplicity  and 
completeness:  simplicity  In  that  the  various  machinations  of  the  reference  tree 
scheme,  although  not  necessarily  intuitively  obvious,  are  nevertheless  uncompli¬ 
cated,  completeness  in  that  all  essential  object  management  functions  (support  for 
side  effects,  multiple  copies,  garbage  collection)  can  be  handled  easily.  The  princi¬ 
pal  liabilities  of  the  reference  tree  scheme  are  In  the  areas  of  reliability  and,  possi¬ 
bly.  efficiency  The  efficiency  of  a  system  using  reference  trees  will  be  a  strong 
function  of  the  shapes  of  reference  trees  that  actually  arise  in  the  system  More 
work  needs  to  be  done  to  determine  the  effect  of  various  scheduling  strategies  on 
the  shapes  in  which  reference  trees  grow. 

To  the  extent  to  wh  ch  reliability  and  efficiency  problems  exist,  many  of  them 
can  be  remedied  by  complicating  the  basic  reference  tree  scheme  Some  such 
modifications  are  suggested  in  Section  5.4  and  in  forthcoming  work  by  Baker[2], 
Ultimately,  reference  trees  may  need  to  be  only  one  of  many  object  management 
strategies  in  a  multiprocessor  system.  Reference  trees  should  be  espec  ially  good 
for  objects  known  only  in  certain  local  areas,  a  category  which  probably  includes 
the  vast  majority  of  objects.  Objects  known  more  or  less  globally  across  the 
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system  might  benefit  (com  being  managed  using  a  different  strategy 

I  inally,  the  usefulness  of  reference  trees  depends  heavily  on  the  conformabil- 
ity  of  a  physical  network  topology  to  the  model  given  in  Section  4  1  Although  net¬ 
works  of  this  sort  have  much  to  recommend  them,  technological  progress  might  ulti¬ 
mately  favor  a  different  kind  of  topology  If  a  network  cannot  be  viewed  as  a  col¬ 
lection  of  nodes,  each  of  whicti  has  "only  a  few"  neighbors,  then  the  usefulness  of 
reference  trees  on  that  network  will  be  greatly  diminished 

8.3:  Performance  of  Reference  Tree  Networks 

This  thesis  can  hardly  be  considered  a  definitive  study  of  the  performance  of 
reference  tree  networks  the  performance  rosults  reported  here  are  too  primitive 
to  form  a  basis  for  comparison  with  Cm*  or  any  other  existing  system  This  is 
partly  because  of  the  paucity  of  applications  studied,  partly  because  of  the  many 
Improvements  that  can  still  bo  made  In  the  MuNet  implementation,  and  partly 
becauso  of  the  large  cost,  on  l  SI- 1  1's,  of  implementing  such  functions  as  message 
input  and  output,  garbage  collection,  and  free  storage  management  About  all  that 
can  be  said  at  this  point  Is  that  'n  some  cases  the  scalability  promises  of  the  archi¬ 
tecture  have  been  realized,  and  in  others  there  is  still  much  work  left  to  do  Many 
different  sets  of  implementation  choices  and  scheduling  strategies  remain  to  be 
explored  Beyond  that,  the  really  important  performance  characteristic  of  a  system 
Is  that  obtained  in  actual  use,  not  that  exhibited  by  toy  benchmarks  In  order  to 
observe  this,  the  system  must  be  made  sufficiently  serviceable  to  attract  ordinary 
users  This  implies  not  only  the  development  of  scheduling  strategies  that  are  at 
least  adequate,  but  also  of  a  congenial  programming  language  and  system  software 
to  support  It  Only  at  this  point  could  a  reference  tree  network  be  considered  truly 
a  going  concern 
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8.4:  Additional  Directions  for  Further  Research 


I  here  is  virtually  no  area  touching  multiprocessor  systems  that  could  not 
benefit  from  additional  research,  herewith  a  brief  list  of  issues  that  relate  to  the 
material  in  this  thesis  The  first  is  the  development  of  applications  tor  reference 
tree  networks  Frameworks  for  general-purpose  applications  such  as  timesharing 
and  simulation,  and  parallel  algorithms  for  special  purposes  such  as  alpha-beta  tree 
search,  or  solution  of  partial  differential  equations,  are  both  needed 

Work  on  some  of  these  applications  may  in  turn  call  Into  question  one  of  the 
design  decisions  underlying  the  construction  of  reference  tree  networks  the 
invisibility  of  network  topology,  interprocessor  communication,  and  monitor  opera¬ 
tions  such  as  garbage  collection  Schemes  tor  making  visible  these  aspects  of 
system  operation  may  help  improve  the  efficiency  of  certain  critical  algorithms, 
although  In  general  such  low-'nvel  details  should  remain  hidden  so  as  not  to  distract 
the  programmer 

Another  important  research  area  may  be  opened  up  by  the  advent  of  easily 
tailored  spociaFpurpose  VI  SI  circuits  Ideally,  in  a  system  whore,  say.  matrix  multi¬ 
plication  is  an  important  operation,  it  should  be  possible  to  hook  up  and  use  a 
special-purpose  matrix  multiplier  without  either  disrupting  operation  of  the  remainder 
of  the  system  or  needing  to  make  extensive  software  changes.  The  specification 
of  constraints  on  the  design  of  both  the  system  and  the  special-purpose  VI  SI  cir¬ 
cuitry  so  that  this  can  be  done  in  a  clean  and  general  fashion  can  be  expected  to 
bo  an  important  research  question  for  the  future  This  can  be  regarded  as  a 
subquestion  of  the  more  general  question  of  how  best  to  incorporate  special- 
purpose  VT  SI  into  general-purpose  systems 

finally,  the  multiprocessor  systems  discussed  in  this  thesis,  although  they  do 
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not  depend  crucially  on  the  assumption,  assume  a  fairly  tight  degree  of  coupling 
between  components  Bcryond  the  merely  physical  degree  of  coupling,  it  is 
assumed  that  all  elements  of  the  system  are  under  the  control  of  one  operating 
authority,  and  that  physical  security  of  the  system  is  not  an  issue  Applying  the 
tecnnology  developed  in  this  thesis  to  more  of  a  "distributed  computing"  context 
where  these  assumptions  may  riot  be  valid  leads  to  whole  new  research  issues 
involving  protection,  awareness  of  physical  system  structure,  and  co-ordination  of 
Independent  authorities  for  name  generation  and  other  functions 

8.5;  Final  Conclusions 

The  marriage  of  VIM,  reference  trees,  and  somo  hardware  has  alroady  pro¬ 
duced  a  system,  the  MuNet,  which  is  capable  of  improving  its  performance  by  mak¬ 
ing  use  of  multiple  processors  Although  much  more  work  remains  to  be  done,  there 
is  reason  to  be  optimistic  that  the  MuNet  and  its  descendants  will  eventually  be 
able  to  exhibit  acceptable  performance  over  a  wide  range  of  tasks.  The  most 
important  feature  of  these  systems,  however,  will  not  be  their  efficiency  but  their 
interface  with  users  and  programs  The  introduction  of  a  parallel  environment 
brings  with  it  a  whole  new  set  of  concerns  for  the  programmer  Our  response  to 
this  situation  must  be  to  remove  as  many  as  possible  of  the  old  concert's,  leaving 
the  programming  job  still  at  a  tractable  level  of  complexity.  Thus  a  good  system 
for  programming  a  multiprocessor  network  must  perform  as  many  housekeeping 
chores  (such  as  communication  and  garbage  collection)  as  possible  automatically, 
freeing  the  programmer's  mind  for  other  worries 

Whether  or  not  reference  tree  networ'  ,  are  the  wave  of  the  future,  the  tech¬ 
nological  arguments  favoring  use  of  multiprocessor  systems  are  irresistible  Some 
methodology  for  using  them,  based  on  ti  n  fundamental  design  considerations  that 
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underlie  reference  tree  networks,  must  be  developed 
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Appendix  A:  The  MuNet  Virtual  Machine 


the  virtual  machine  supported  by  the  MuNet  is  in  some  ways  an  elaboration, 
and  In  other  ways  a  subset,  of  the  VIM  machine  discussed  in  Chapters  2  and  3  of 
this  thesis  The  differences  reflect  the  pragmatics  of  optimizing  use  of  the  MuNet 
hardware  (a  collection  of  l  SI- 11  microprocessors),  and  of  decisions  Intended  to 
expedite  the  construction  of  the  initial  implementation  The  MuNet  also  bears  the 
scars  of  having  been  produced  by  an  evolutionary  process,  several  aspects  of  the 
MuNet  would  be  constituted  differently  In  any  re-implementation  The  purpose  of 
this  appendix  is  to  document  the  current  MuNet  virtual  machine,  and  give  some 
examples  of  both  good  and  bad  ideas  In  reference  tree  network  architecture  The 
appendix  is  not  designed  to  be  self-contained,  but  should  be  intelligible  in  combina¬ 
tion  with  Chapters  ?  and  3  of  this  thesis  further  details  about  the  MuNet  exist  as 
Internal  implementation  notes[39] 

A.1:  Object  References 

On  the  MuNet,  an  object  reference  occupies  a  16-bit  word*  Ordinary  object 
references  are  pointers  to  words  in  memory  and  hence,  on  the  LSI-11,  are  always 
even  numbers  This  fact  opens  the  door  for  the  use  of  odd  numbers  as  special 
"reserved"  object  references  An  odd  number  appearing  where  a  reference  is 
expected  is  not  a  reference  to  any  particular  object,  but  is  a  distinguishable 
entity,  differentiable  from  all  other  odd  numbers  and  from  all  ordinary  object 

"This  is  a  local  name,  valid  only  on  the  processor  where  the  reference  exists; 
It  does  not  imply  any  limitation,  such  as  ?’ °,  on  the  number  of  objects  that  could 
exist  in  the  network  It  is  thus  incorrect  to  say  that  the  MuNet  has  an  "address 
space"  of  only  16  bits  Taking  into  consideration  the  size  allotted  in  the  MuNet  for 
global  names  of  objects,  the  MuNet  "address  space”  can  be  considered  as  large 
as  40  bits 
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references  In  certain  contexts,  specific  odd  numbers  have  special  meanings  to 
the  system 

As  In  VIM,  each  ordinary  MuNet  object  reference  has  an  associated  text,  which 
may  be  accessed  using  suitable  monitor  calls.  In  addition,  a  MuNet  object  refer¬ 
ence  has  other  attributes,  independent  of  its  toxt  Since  the  only  facilities  that 
exist  for  changing  an  object  oj>erate  by  changing  its  text,  these  other  attributes 
must  be  determined  when  an  object  is  created,  and  cannot  be  changed  thereafter, 
[very  MuNet  object  has  two  such  attributes.  The  first  is  its  type,  which  in  the 
MuNet  is  a  separate  property  of  an  object,  not  simply  a  distinguished  reference 
within  its  text,  as  it  Is  in  VIM  (see  Figure  2.3).  It  follows  that  every  MuNet  object 
has  a  type,  whether  or  not  the  objoct  is  ever  used  as  a  target  object  in  an  event, 
and  that  an  object’s  type  must  be  specified  when  it  is  created. 

Experience  with  the  MuNet  shows  that  this  is  an  Inferior  way  of  handling 
types  Not  only  does  it  rob  the  user  of  the  power  to  change  an  object’s  type,  but 
the  special  treatment  for  types  complicates  various  sections  of  monitor  code  and 
increases  the  space  and  time  overhead  for  object  management  Finally,  it  forces 
objects  to  have  a  type  even  where  (as  In  the  case  of  purely  "data"  objects)  the 
VIM  type  concept  is  not  exactly  what  is  desired,  leading  to  further  wastage  of  time 
and  space 

The  MuNet  distinguishes  several  classes  of  objects  according  to  their  types 
Genera/  objects  are  most  like  the  target  object  shown  In  Figure  2.3.  the  type  of  a 
general  object  is  a  reference  to  another  object.  This  object,  however,  cannot  be 
another  general  object,  but  must  bo  a  type  object,  whose  type  is  the  reserved 
integer  5.  If  the  "closure"  aspect  of  general  objects  Is  not  needed,  a  target 
object  may  bo  a  function  object,  whose  type  Is  the  reserved  Integer  3.  When  a 
target  object  Is  a  function  object,  the  machine  code  to  be  executed  is  found 
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directly  In  the  text  of  the  function  object  Thus  a  function  object  is  analogous  to 
a  VIM  target  object  whose  type  is  a  reference  to  Itself.  Finally,  the  reserved 
integer  l  may  be  used  as  an  object's  type  to  denote  a  data  object  which  should 
never  be  used  as  a  target  object.  No  other  odd  integer  may  be  used  as  a  type; 
these  other  numbers  are  reserved  for  use  by  the  Implementation  to  distinguish 
objects  from  events  and  from  other  entities  used  internally. 

The  other  permanent  attribute  of  an  object  is  a  Boolean  device  1 lag  If  an 
object's  device  flag  is  set.  the  text  of  the  object  cannot  be  moved  from  the  loca¬ 
tion  where  the  object  was  created  Therefore,  an  event  that  requests  access  to 
the  text  of  such  a  "device  object"  is  forced  to  execute  on  the  processor  where 
that  text  is  the  text  cannot  be  moved  to  where  the  event  is  Device  objects 
are  intended  to  model  peripheral  devices  which  exist  on  particular  processors  An 
event  whose  execution  directly  involves  transfers  of  data  to  or  from  a  peripheral 
must  execute  on  the  processor  where  the  peripheral  is  If  the  target  object  used 
for  final  transactions  with  the  peripheral  is  installed  as  a  "device"  on  the  appropri¬ 
ate  processor,  this  outcome  will  be  ensured  (since  gtext  access  to  the  target 
object  is  requested  in  the  course  of  beginning  execution  of  an  event)  A  mechan¬ 
ism  for  creating  device  objects  will  be  discussed  later 

Although  devices  are  quite  a  "dirty"  mechanism,  they  are  extremely  useful  any 
time  it  is  desired  to  force  execution  of  a  particular  event  in  a  particular  place 
This  is  the  case  not  only  when  Interacting  with  peripherals,  but  also  for  perfor¬ 
mance  monitoring  and  other  kinds  of  intervention  in  the  activities  of  the  MuNet. 
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A. 2:  Object  Texts 

An  object  text  within  a  MuNet  processor  is  represented  as  a  series  of  words 
in  memory  A  text  begins  with  a  header  word,  followed  by  a  sequence  of  r  words 
containing  object  references,  followed  by  w  words  containing  uninterpreted  binary 
data  (see  f  igure  A  1).  The  low-order  byte  of  the  header  word  contains  the  quan¬ 
tity  r.  while  the  high-order  byte  contains  the  quantity  r*w  Since  this  quantity 
must  be  representable  in  a  byte,  the  length  of  object  texts  may  not  exceed  255 
words  Larger  aggregates  of  data  must  be  represented  ds  several  objects  joined 
together  using  object  references 

header  word 

r  references 

w  words 

f  igure  A  1  Format  of  MuNet  object  texts 

If  an  object  is  a  function  or  type  object,  it  Is  presumed  to  contain  executable, 
position-independent  POP- 1  1  machine  code  starting  at  the  beginning  of  the  "binary 
words"  portion  of  Its  text.  Control  will  be  transferred  to  the  beginning  of  this  code 
whenever  the  object's  code  is  invoked  in  executing  an  event  The  ability  to 
Include  references  along  with  actual  machine  code  in  an  executable  text  provides 
a  useful  way  of  making  available  references  (e.g..  to  other  parts  of  the  program) 
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which  may  be  needed  during  event  execution 


A. 3:  Events 

An  event  has  the  same  format  as  an  object  text  It  begins  with  a  header 
word,  followed  by  a  series  of  references  (of  which  the  first  is  viewed  as  the  refer¬ 
ence  to  the  target  object),  followed  by  a  (possibly  null)  series  of  binary  words 
When  an  event  is  to  be  executed,  a  pointer  to  the  event  is  placed  in  the  PDP-11 
register  r 0  A  pointer  to  the  text  of  the  target  object  is  then  placed  in  rl  (note 
this  is  not  necessarily  the  text  that  contains  the  machine  code  to  be  executed) 
Next,  control  is  transferred  to  the  executable  code  associated  with  the  target 
object 

While  executing,  an  event  can  call  on  monitor  services  via  jsr  or  jmp  instruc¬ 
tions  through  transfer  vectors  at  absolute  locations  in  low  core  The  calls  available 
will  bu  enumerated  presently,  some  further  conceptual  differences  between  VIM 
and  the  MuNet  should  be  discussed  first 

In  VIM,  every  event  is  created  by  a  newev  request,  eventually  executes  suc¬ 
cessfully,  and  then  becomes  inactive  If  an  event  causes  additional  events  to 
further  the  progress  of  a  computation,  it  must  create  each  such  event  by  means  of 
another  call  to  newev  It  is  possible  to  use  this  mode  of  operation  when  program¬ 
ming  the  MuNet,  but  for  the  sake  of  efficiency,  an  alternative  has  been  made  avail¬ 
able 

Many  events  cause  exactly  one  event  to  carry  on  the  computation;  /.e..  In 
most  cases,  the  thread  of  event  causality  is  basically  linear,  with  relatively  few 
forks  and  joins  In  this  situation,  it  is  desirable  to  reuse  the  storage  allocated  to 
the  current  event,  rather  than  go  through  the  overhead  of  throwing  it  away  and 
allocating  a  new  event.  Thus  if  execution  of  an  event  on  the  MuNet  ends  with  a 


Section  A, 3  Events 


219 


nextev  request,  rather  than  a  done,  the  current  event  is  not  thrown  away,  but 
remains  active  and  is  enqueued  for  re-executiori.  Unless  it  was  the  user’s  intention 
to  create  an  endless  loop,  he  will  presumably  have  modified  the  contents  of  the 
event  so  that  it  now  calls  for  the  next  step  in  the  computation,  r.e.,  he  will  have 
written  into  it  wtiat  would  otherwise  have  been  written  into  an  event  created  using 
newev,  had  that  alternative  been  adopted 

A  few  additional  comments  about  the  nextev  mechanism  are  appropriate  First, 
writing  into  the  current  event  must  be  considered  a  side  effect  (in  the  sense  that 
It  would  set  the  blackboard-interpreter  side-effect  flag  *  to  true),  and  consequently 
must  be  done  after  all  resource  requests  have  been  made  if  an  aborted  event 
has  been  modified,  re-execution  of  the  event  Is  not  likely  to  yield  correct  results 
Second,  the  nextev  mechanism  is  not  inconsistent  with  forks  It  is  entirely  reason¬ 
able  to  create  sevoral  new  events  using  newev  and  also  reuse  the  current  event 
by  calling  nextev  Third,  an  evxpnd  primitive  (described  below),  analogous  to 
objxpnd,  is  provided  for  handling  situations  where  reusing  the  current  event  may 
require  increasing  its  si/e 

Another  note  regarding  events  in  the  MuNet  —  every  event  has  Implicitly  asso¬ 
ciated  wit*-;  it  an  operating  system  object,  accessible  via  the  getos  and  setos  moni¬ 
tor  calls  The  idea  behind  operating  system  objects  is  to  make  available  the  kinds 
of  capabilities  associated  with  pro>  ess  objects  in  chapter  3  of  this  thesis;  how¬ 
ever,  none  of  these  functions  is  currently  implemented  on  the  MuNet 

A  second  difference  between  VIM  and  the  MuNet  is  that  in  VIM,  gtext  and 
locktext  requests  serve  only  to  reserve  access  rights  to  object  texts,  no  values 
are  returner!  When  an  executing  event  actually  accesses  data  in  an  object  text, 
it  does  so  by  supplying  a  reference  to  the  object  along  with  identification  of  a  slot 
in  the  object  text  The  implementat.on  is  then  responsible  for  finding  the  text  in 
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core  and  performing  the  desired  access  for  the  sake  of  efficiency,  MuNet  gtext 
and  locktext  requests  return  a  pointer  to  the  relevant  text,  which  may  then  be 
accessed  directly  using  I’D!’- 1  1  indexed  addressing  modes  I  his  pointer  is  only 
good  for  the  remainder  of  the  current  event  execution,  since  the  object  text  might 
be  moved  before  any  subsequent  execution  Any  subsequent  execution  wishing  to 
access  the  object  will  have  to  issue  another  gtext  or  locktext  request  in  any 
case,  and  should  use  the  text  pointer  returned  from  that  request 

Unfortunately,  the  objxpnd  request,  which  increases  the  si/e  of  the  area  allo¬ 
cated  to  hold  a  text,  may  need  to  move  the  text  in  order  to  fulfill  its  mission  thus 
objxpnd  returns  a  new  text  pointer  which  obsoletes  any  text  pointer  previously 
obtained  for  the  object  during  the  same  event  execution  fhe  resulting  potential 
for  confusion  is  mitigated  somewhat  by  the  infrequency  of  objxpnd  requests,  but  it 
is  fairly  clear  that,  in  the  presence  of  appropriate  hardware  support  for  accessing 
texts,  a  policy  of  not  explicitly  releasing  text  pointers  is  simpler  and  cleaner 

A  third  departure  from  VIM  results  from  the  nature  of  the  LSI- 1  1  hardware  base 
for  the  MuNet  In  VIM,  the  legality  of  various  operations  (such  as  aborting  an 
event)  is  contingent  on  the  setting  of  the  side-effect  flag  »,  indicating  whether  any 
side  effects  have  been  performed  by  the  event  in  question  The  lack  of  memory 
management  hardware  on  the  MuNet  LSI- 1  1 '»  precludes  any  attempt  to  maintain 
such  a  flag  automatically  It  would  be  possible  to  construct  a  monitor  call  setting 
the  side-effect  flag  the  user  could  then  invoke  this  call  upon  performing  a  side 
effect  However,  this  approach  was  not  taken,  and  in  fact  no  side-effect  flag  is 
explicitly  maintained  for  events  in  the  MuNet  this  means  that  the  state  of  an 
event’s  side-effect  flag  cannot  be  used  to  determine  the  legality  of  aborting  it 
MuNet  programs  should  st  !•  follow  the  same  rules  as  if  the  side-effect  flag  were 
maintained  and  checked,  though,  thus  it  is  still  unwise,  for  example,  to  make  a 
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gtext  request  after  having  performed  a  side  effect  the  MuNet  monitor  cannot  sig¬ 
nal  an  error  if  this  is  done,  since  it  has  no  knowledge  of  whether  a  side  effect  has 
been  performed  Rather,  this  discipline  must  be  followed  by  the  user,  or  he  risks 
obtaining  incorrect  results 

The  need  to  be  able  to  abort  events  Is  as  real  on  the  MuNet.  however,  as  It  is 
in  VIM  Since  there  Is  no  explicit  side-effect  flag  on  the  MuNet  to  indicate  whether 
aborting  an  event  is  permissible,  events  on  the  MuNet  can  only  be  aborted  at 
places  where  it  can  bo  Inferred  that  the  side-effect  flag,  had  it  been  maintained, 
would  be  false  This  inference  can  be  made  any  time  an  event  makes  a  request 
which,  according  to  the  rules  of  VIM.  is  only  legal  when  the  side-effect  flag  is  false 
Thus  events  can  be  aborted  only  while  trying  to  mako  such  requests 

A. 4:  Monitor  Calls 

A  listing  of  MuNet  system  calls  follows  In  each  case,  a  schematic  form  of  the 
system  call  (eg  .  gtype(ref)  raturns(ref ))  is  given,  along  with  the  actual  PDP-11 
calling  sequence  and  a  description  of  the  effoct  of  the  call 

gtypa(ref)  returns(fypo) 

mov  Uraf.type 

returns  (moved  into  type)  the  type  of  the  object  referenced  by  ref 
gtext(ref)  returns(fe«rfpfr) 

mov  ref.rO 

jar  pc,##gtext.v 

mov  rl.fexfpfr 

obtains  sharable  (read-only)  access  to  the  text  of  the  object  referred  to  by 
ref.  and  returns  a  pointer  to  that  text  gtext  may  not  be  called  following  a 
side  effect. 
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locktext(ref )  r«turns(fn»  tptr  ) 


ret.rO 

pc,9#locMaxt  v 
rl  ,fe*  tptr 


mov 

J*' 

mov 

like  gtext  in  every  respect,  except  that  it  obtains  non-shared  (read/write) 
access  to  the  text  in  question 


newobj(fype,  si.’o)  returns(ref  ,fe» fpfr ) 


mov 

fype.rO 

mov 

s//e,r1 

Jsr 

pc,9#newobj  v 

mov 

rO.ref 

mov 

r1,fe*fpfr 

creates  an  object  with  type  fype  and  sr/e  words  of  text  (not  counting  the 
header  word)  A  reference  to  the  newly  created  object  is  returned,  along 
with  a  pointer  to  the  new  object's  text  newobj  may  not  be  called  following 
a  side  effect 


objxpnd(ref  ,s//e)  returns(fe  *  fpf  r ) 


mov 

rof.rO 

mov 

s/ze,r  1 

)*' 

pc,9#objxpnd  v 

mov 

rl  .foxfpfr 

ensures  that  the  area  allocated  to  storage  of  the  text  of  the  object 
referred  to  by  ref  is  at  least  large  enough  to  support  a  text  of  length  s//e 
words  (not  counting  the  header  word)  If  the  area  allocated  to  the  object 
text  is  not  large  enough,  the  text  is  moved  to  a  new,  suitably  large  location 
the  fe*fpfr  returned  is  a  pointer  to  the  text  affer  any  required  relocation, 
and  supersedes  any  fexfpfr  previously  obtained  for  the  same  object  during 
the  same  event  execution  *  On  the  presumption  that  a  caller  of  objxpnd 
intends  to  modify  at  least  the  header  word  of  the  text,  objxpnd  performs  an 
Implicit  locktaxt  on  the  object  objxpnd  may  not  be  called  following  a  side 
effect  However,  since  objxpnd  does  not  actually  modify  the  text,  calling  It 
Is  not  considered  a  side  effect 

*A  better  approach  would  bo  to  have  objxpnd  never  return  a  fexfpfr,  but 
instead  abort  the  requesting  event  if  the  text  is  moved  The  text  having  been 
moved  to  a  more  spacious  location,  the  objxpnd  would  presumably  succeed  upon 
the  next  execution  of  the  event  Aborting  the  event  when  a  text  is  moved,  how¬ 
ever,  forces  all  fe*tpfrs  used  by  the  event  to  be  re-obtained,  thus  avoiding  the 
possibility  of  needing  to  discard  a  fenfpfr  because  of  a  subsequent  objxpnd.  and 
the  hazards  of  negligently  continuing  to  use  such  a  fexfpfr 
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newev(nwords)  return*(evp/r) 

mov  nwords.rf 

)*r  pc,l#n«wev  v 

mov  rl.evpfr 

creates  a  new  event  nwords  words  tong  and  returns  a  pointer  to  it  in  evptr 
The  new  event  will  ho  released  for  execution  only  if  execution  of  the 
current  event  terminates  successfully  As  with  a  newobj  request,  a  newev 
request  may  not  be  made  following  a  side  effect 


evxpnd(nwords)  returns(eirptr) 

mov  otvo/ds.rl 

Jsr  pc,S#evxpnd  v 

mov  r  1  ,evplr 

evxpnd  is  like  objxpnd,  except  that  it  operates  on  the  currently  executing 
event  rather  than  on  an  object  text  evxpnd  is  called  when  reuse  of  the 
current  ovont  is  contemplated,  and  assurance  is  dosired  that  at  least 
nworcls  words  of  event  space  are  available  to  hold  references  and  binary 
words  like  objxpnd.  evxpnd  is  not  considered  a  side  effect  (does  not 
actually  modify  the  contents  of  the  event),  may  not  be  called  following  a 
side  effect,  and  may  relocate  the  ovent  to  a  more  spacious  location  Thus 
after  a  call  to  evxpnd.  the  returned  e vptr  should  be  used  in  lieu  of  any  pre¬ 
viously  obtained  pointer  to  the  current  event 


nextevO 


mov  •#nextev.v,pc 

terminates  (successfully)  execution  of  the  current  event  and  re-enqueues  it 
on  the  event  list  Any  other  events  or  objects  created  are  also  formally 
Installed 


dona() 


rts  pc 

like  nextev,  but  does  not  re-enqueue  the  current  event  Because  of  Its  use 
of  a  return  address  on  the  processor  stack,  the  stack  should  be  popped 
back  to  its  state  when  execution  of  the  current  event  began  before  done  is 
Invoked 
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getosO  raturns(osref ) 


Jtr  pc,*#gato«_v 

mov  rO.osref 

returns  a  reference  to  the  operating  system  object  associated  with  the 
current  event  This  request  can  never  cause  an  event  to  be  aborted  and 
may  be  executed  at  any  time. 

setos (osref ) 

mov  osref, rO 

jar  pc,##»eto»  v 

changes  the  operating  system  object  of  the  currently  executing  event  to 
osref  I  his  is  considered  a  side  effect  and  should  not  be  done  if  the  event 
might  subsequently  bo  aborted  Note  that  the  operating  system  object  for 
any  newly  created  events  (made  using  newev)  will  be  that  operating  sys¬ 
tem  object  in  effect  at  the  emf  of  the  execution  of  the  creating  event 


A  few  other  monitor  calls  exist  also,  allowing  access  to  internal  monitor  variables, 
interaction  with  the  UNIX  development  environment,  and  implementation  of  other 
specialised  functions  These  calls  are  not  intended  for  use  by  ordinary  users,  but 
aid  in  the  construction  of  special  actors  performing  various  system  initialisation  and 
monitoring  functions 

A.S:  Special  Objects 

When  the  MuNet  is  in  normal  operation,  two  kinds  of  special  objects  exist 
within  it  processor  objfM'ts  and  a  system  object  There  is  one  system  object  for 
the  entire  MuNet,  which  may  he  invoked,  as  a  target  object,  to  obtain  information 
and  perform  functions  of  system-wide  significance  Among  the  kinds  of  events  that 
the  MuNet  system  object  will  respond  to  are  (using  S  to  denote  a  reference  to  the 
system  object) 
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(S  1  C)  causes  the  event  (C  X),  where  X  is  a  reference  to  an  array  of  refer¬ 
ences  to  all  the  processor  objects. 

(S  3  C)  causes  one  event  ( C  P)  tor  each  processor  object  P. 

Other  calls  to  the  MuNet  system  object  allow  I/O  to  the  UNIX  console  from  which 
the  MuNet  was  invoked,  and  provide  mechanism  for  processor  objects  to  inform  the 
system  object  of  their  existence 

In  a  fully  operational  MuNet.  there  Is  one  processor  object  for  each  physical 
processor  I  ach  processor  object  is  a  "device"  Installed  on  the  processor  it 
corresponds  to  Among  the  kinds  of  events  that  a  processor  object  P  will  respond 
to  are 

(P  1  X  C)  causes  the  event  (C  Y)  where  Y  is  a  copy  of  the  object  X  (same  type, 
same  text  contents)  installed  as  a  device  on  the  processor  whose  proces¬ 
sor  object  P  is. 

(P  3  C.)  causes  the  event  (C  X)  whore  X  is  a  new  object  whose  text  contains 
various  pieces  of  status  information  about  the  processor  associated  with  P. 

A. 6:  Summary 

This  appendix  has  given  an  overview  of  the  virtual  machine  supported  by  the 
current  MuNot  implementation.  Generally  speaking,  the  VIM  virtual  machine 
described  In  the  text  of  the  thesis  should  be  regarded  as  superior  to  the  MuNet 
virtual  machine  where  their  features  differ,  the  purpose  of  describing  the  MuNet 
virtual  machine  here  has  been  to  expose  some  of  the  Influences  on  and  alterna¬ 
tives  to  VIM,  and  to  show  the  elaboration  of  some  of  the  VIM  concepts  down  to  a 
more  concrete  and  practical  level 
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The  purpose  of  this  appendix  is  to  demonstrate)  that  the  reference  tree 
membership  protocol  described  in  Section  fa. 3  performs  the  functions  claimed,  viz., 
that  it  prevents  reference  trees  from  becoming  disconnected,  prevents  cycles  from 
forming  In  them,  and  Is  not  prey  to  any  other  sort  of  error  condition  As  a  side 
effect,  perhaps  the  demonstration  will  shed  a  little  additional  light  on  the  workings 
of  the  reference  tree  mechanisms. 

The  membership  protocol  Is  specified  by  the  list  of  state  transitions  in  Table 
ft  9,  plus  several  restrictions  not  explicitly  stated  in  the  table  In  this  appendix,  it 
will  bt«  convenient  to  refer  to  various  of  the  state  transition  rules  in  Table  fa. 9. 
The  notation 


old-state  :  message-received  /  message-sent  :  new-state 

will  be  used,  so.  for  example.  "X:R  +  /i*:S"  refers  to  a  transition  that  happens  when 
a  processor  in  state  X  receives  an  R*  message,  changing  to  state  S  and  emitting 
an  l  ♦  message  back  to  the  sender  of  the  R*  message  Spontaneous  transitions 
have  a  null  message-rtH'eived  field,  and  a  null  message-sent  field  indicates  a  transi¬ 
tion  not  accompanied  by  any  output  Thus  X:/:N  denotes  a  spontaneous  transition 
from  state  X  to  state  N  which  results  in  no  output 

There  are  two  kinds  of  Important  restrictions  on  the  applicability  of  the  spon¬ 
taneous  transitions  listed  in  Table  fa  9  One  applies  only  to  the  transition  M:/-:X?, 
by  which  a  processor  ieaves  the  reference  tree  A  processor  is  allowed  to  make 
this  transition  only  if  its  state  for  every  other  link  attached  to  it  is  either  N  or  N? 
This  means,  among  other  things,  that  the  processor  making  the  transition  must  be  a 
leaf  node  of  the  reference  tree  At  the  time  when  the  transition  is  made,  the  proc- 
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essor’s  other  link  states  of  N  and  N?  must  be  changed  to  X  and  X?,  respectively. 

the  other  kind  of  restriction  applies  to  the  transitions  N:/:X,  N?:/:X?,  X:/:N, 
and  X?:/:N?  The  first  two  of  these  exist  only  to  represent  the  state  changes  on 
other  links  that  are  associated  with  a  processc  leaving  the  reference  tree,  as  dis¬ 
cussed  In  the  previous  paragraph,  and  should  not  be  used  at  any  other  time  Thus, 
although  the  transitions  look  spontaneous,  they  are  in  fact  only  used  when  other 
activities  on  the  processor  cause  it  to  leave  the  reference  tree  Similarly,  the 
transitions  X:/:N  and  exist  only  to  handle  state  changes  that  must  occur 

when  a  processor  Joins  a  reference  tree  Since  it  is  always  the  receipt  of  a  mes¬ 
sage  that  causes  a  processor  to  Join  a  reference  tree,  there  is  a  sense  In  which 
these  two  transitions  are  never  "spontaneous  ” 

Showing  the  correctness  of  the  membership  protocol  divides  into  two  tasks 
determining  some  local  properties  of  the  protocol  as  it  applies  to  a  microcosm  of 
just  two  nodes  and  one  link,  and  generalizing  from  these  to  prove  global  properties 
of  entire  reference  trees  An  assumption  that  we  shall  make  throughout  is  that 
only  one  thing  (a  spontaneous  transition  or  absorption  of  a  message)  happens  at  a 
time  In  practice,  this  means  only  that  each  processor  must  act  as  an  arbiter 
among  messages  arriving  at  it  over  different  links,  and  process  them  sequentially 
The  need  to  ensure  more  global  sequentiality  can  be  circumvented  by  noting  that 
the  effects  of  state  changes  at  processors  are  strictly  local  Thus  simultaneous 
events  at  different  processors  can  he  r.onsidnrftd  to  have  happened  sequentially, 
in  any  order,  for  the  purposes  of  this  appendix 
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B.  1 :  Local  Properties  of  the  Membership  Protocol 

Local  properties  of  the  membership  protocol  are  the  following; 

e  closure  —  all  configurations  reachable  from  any  possible  starting  configuration 
should  be  possible  to  handle  following  the  rules  of  the  protocol;  no  proces¬ 
sor  should  ever  receive  a  message  that  it  cannot  handle  in  its  current  state 

•  consistency  all  quiescent  configurations  (/.e.,  configurations  in  which  no 
messages  aro  in  transit)  reachable  from  any  possible  starting  conhguration 
should  show  the  desired  relationship  between  processor  states;  for  exam¬ 
ple,  we  would  not  like  to  see  a  quiescent  conhguration  in  which  two  proces¬ 
sors  each  thought  they  were  masters  of  the  same  link. 

From  our  local  examination  of  the  membership  protocol  we  should  also  like  to 
abstract  a  few  properties  useful  in  showing  global  properties  of  the  protocol. 
These  properties  deal  with  the  relationship  of  the  membership  status  (r.e.,  in  or  out 
of  a  reference  tree)  of  pairs  of  adjacent  nodes  to  th«»  membership  status  of  the 
link  connecting  them 

As  discussed  in  Section  5.3,  every  node,  or  processor,  has  (conceptually)  a 
processor  stale  for  each  reference  tree  This  state  can  be  either  “in  the  refer¬ 
ence  tree"  or  "not  in  the  reference  tree  ”  The  processor  state  is  manifested  in 
the  values  of  the  processor's  link  state  for  each  link  attached  to  the  processor, 
such  that  if  a  processor  is  “not  in"  a  particular  reference  tree,  each  of  its  link 
states  for  that  tree  will  be  either  X  or  X?.  Conversely,  if  the  processor  is  “in"  the 
tree,  each  of  its  link  states  for  that  tree  will  be  some  state  other  than  X  or  X? 
Thus  it  is  easy  to  determine  whether  a  processor  is  a  member  of  a  particular  refer¬ 
ence  tree,  examination  of  any  of  that  processor's  link  states  for  that  tree  will 
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yield  the  answer. 


It  is  more  difficult  for  a  processor  to  find  out  whether  a  link  that  connects  to  it 
is  a  member  of  a  particular  reference  tree  In  general,  knowing  the  link  state  of 
that  processor  for  that  link  is  not  sufficient  tor  an  unambiguous  determination 
(exceptions  are  states  M  and  Nl,  which  always  mean  the  link  is  in  the  reference 
tree,  and  states  X,  X"7,  N.  and  N"7,  which  always  mean  it  is  not)  In  fact,  even 
knowing  the  link  state  at  both  ends  of  the  link  will  not  always  be  enough  to  decide 
whether  the  link  is  in  the  reference  tree.  In  addition  to  the  link  states,  the 
queues*  of  pending  messages  (messages  sent  but  not  yet  received)  on  the  hnk 
must  be  taken  into  consideration  to  determine  with  certainty  whether  a  particular 
link  is  a  member  of  a  particular  reference  tree 

Thus  in  order  to  completely  specify  the  condition  of  a  link  between  processors 
A  and  B,  for  a  particular  object's  reference  tree,  four  pieces  of  Information  are 
needed  the  link  states  of  processors  A  and  B  for  that  link  with  respect  to  that 
object,  and  the  queues  of  messages^  sent  but  not  yet  received  from  A  to  B  and 
from  B  to  A  In  our  discussions,  we  shall  represent  the  condition  of  a  link  as  fol¬ 
lows 


(A-state  A-fo-B-gi/eoe  B-sfafe  B-fo-A-gi/eoe) 
Thus  the  meaning  of  the  link  condition 

(M?  (R*  A  )  M?  1  0) 


*The  reader  is  reminded  that  messages  on  a  link  must  be  received  in  the  same 
order  as  they  were  sent,  In  order  for  the  protocols  to  work  properly.  Consequently, 
the  notion  of  a  "queue"  of  messages  on  each  link  is  an  accurate  one. 

^These  messages  may  be  any  of  R*.  L»,  L-,  ♦,  A»,  A-.  or  LN-,  In  other  words, 

any  of  the  messages  mentioned  in  Table  6.9. 
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is  that  the  link  state  ot  processor  A  tor  this  link  is  M?  and  the  link  state  of  proces¬ 
sor  B  is  N?1,  that  an  A  followed  by  an  message  has  been  sent  by  A  to  B  but 
neither  has  yet  been  received  by  B,  and  that  no  unreceived  messages  have  been 
sent  by  B  to  A.  Since  interprocessor  links  in  reference  tree  networks  are  symmet¬ 
rical,  the  link  condition 

(A-sfafe  A-fo-B-guoue  B-sfa/e  B-fo-Agt/ei/e) 

is  for  all  practical  purposes  equivalent  to 

(B-sfafe  B-fo-A-g</«H/e  A -s/a/e  A-fo-B-gt/eue) 

in  that  any  comment  that  may  be  made  about  one  of  thorn  is  valid,  mulatis 
mutandis,  about  the  other  In  this  appendix,  we  shall  follow  the  practice  of  allow¬ 
ing  one  of  these  versions  to  stand  for  both,  rather  than  taking  the  extra  space  to 
list  both  versions  of  all  asymmetrical  link  conditions 

Obviously  an  infinite  number  of  different  link  conditions  are  possible,  since  each 
message  queue  could  bo  an  arbitrarily  long  string  Therefore,  it  would  seem  neces¬ 
sary  to  formulate  a  rule,  or  algorithm,  that  could  be  applied  to  a  particular  link  con¬ 
dition  to  determine  whether  that  link  should  be  considered  a  member  of  the 
relevant  reference  tree  Unfortunately,  it  is  troublesome  to  design  a  concise  rule 
for  this  purpose,  so  an  alternative  route  is  taken  in  this  appendix. 

At  the  birth  of  a  system,  when  we  are  entitled  to  beliovo  that  no  inconsisten¬ 
cies  exist  and  no  messages  are  already  flowing,  every  link  will  be  in  one  of  the  fol¬ 
lowing  conditions: 

(X  0  X  ())  (X  OHO)  (N()  NO)  (L  ()  L  ())  (S  ()  M  ()) 


Section  B  I:  local  Properties  of  fhe  Membership  Protocol 


231. 


Thus  the  only  link  conditions  that  will  really  be  ot  interest  to  us  are  those  that  can 
be  reached  trom  this  initial  set  by  following  the  state  transition  rules  spelled  out  In 
Table  5  9  This  closed  subset  of  the  entire  set  of  link  conditions  is  the  only  one  in 
which  we  are  concerned  with  being  able  to  determine  membership  of  links  in  refer¬ 
ence  trees.  Although  this  subset  is  still  infinite,  we  can  group  its  elements  into  a 
finite  number  of  useful  categories  by  making  one  simple  observation  Since  sending 
a  local  name  never  has  any  effect  on  the  sender's  state,  and  receiving  any  series 
of  consecutive  repetitions  of  a  local  name  has  the  same  effect  on  the  receiver’s 
state  as  receiving  just  one.  we  can  use  the  symbol  IN  in  a  message  queue  to 
denote  not  just  a  single  local  name,  but  any  sequence  of  one  or  more  repetitions  of 
the  local  name  As  a  result,  transitions  involving  the  receipt  of  a  local  name  by  a 
processor  must  be  performed  in  each  of  two  ways:  by  removing  the  IN  from  the 
input  queue,  the  proper  action  if  the  IN  represented  only  one  local  name,  and  by 
leaving  the  LN  in  ttie  input  queue,  the  proper  action  if  it  represented  several 
repetitions  of  the  local  name 

With  the  simplification  of  letting  one?  instance  of  the  symbol  LN  represent  any 
number  of  repetitions  of  the  local  name,  the  number  of  link  conditions  accessible 
from  our  initial  set  becomes  finite  A  mechanically  generated  list  of  all  these 
accessible  link  conditions  is  given  below  in  Table  B.l. 

In  the  table,  each  link  condition  is  given  with  a  reference  number,  an  indication 
of  whether  the  link  in  question  should  be  considered  a  member  of  the  reference 
tree,  and  the  reference  numbers  of  all  link  conditions  that  could  be  successors  of 
the  current  one  (r.e.,  which  could  be  derived  from  it  by  applying  some  transition  in 
Table  6  9)  The  status  of  the  link  as  being  in  or  out  of  the  reference  tree  has 
been  chosen  so  that 
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•  the  status  remains  Invariant  trow  a  link  condition  to  any  successor  except 


that 


•  when  one  of  the  processors  undergoes  one  of  the  transitions 
X:FU/l  +  :S  or  X?:LN/: N!,  the  link  (which  should  not  previously  have 
been  In  the  tree)  joins  the  tree,  and 

•  when  the  link  and  both  processors  are  in  the  tree  and  one  of  the 
processors  undergoes  the  transition  M:/  :X?,  leaving  the  tree,  the 
link  also  leaves  the  tree,  and 

•  the  link  is  not  In  the  tree  for  the  quiescent  conditions 

(X  ()  X  ())  (X  ()  N  ())  (MONO)  (L  ()  L  ()) 

but  is  in  the  tree  for  the  quiescent  condition  (S  ()  M  ()). 

That  the  assignment  of  link  status  (in  or  out  of  the  reference  tree)  to  link  condi¬ 
tions  in  Table  B  1  has  these  properties  has  been  verified  mechanically;  the  reader 
Is  invited  to  peruse  the  table  to  convince  himself  of  this  and.  perhaps,  gain  further 
insight  into  our  approach.  The  reader  should  bear  in  mind,  when  following  the  rela¬ 
tionships  between  link  conditions  and  their  successors,  that  only  one  of  the  link 
conditions  belonging  to  each  symmetrical  pair  is  listed.  Thus  the  change  from  a  link 
condition  to  its  successor  in  Table  R  1  may  involve  not  only  a  state  transition  at  a 
processor,  but  also  a  reversal  of  the  roles  of  processors  A  and  B. 
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Table  B.T:  Member  ship  Protocol  Link  Conditions 


Is  Link 

Number  link  Condition 

In  Tree?  Successor s 

1. 

(L  ()  L  0) 

No 

2.  4 

2. 

(L  ()  L  UN)) 

No 

1.  2.  3.  5.  15 

3 

(L  UN)  L  UN)) 

No 

2.  3,  8 

4 

(L  ()  L?  (  )) 

No 

6,  15,  1  7,  225 

6 

(L  ()  L7  (-  IN)) 

No 

4,  ft 

i,  7,  8,  18 

6 

(L  ()  L?  UN  -)) 

No 

6,  16,  19,  226 

7. 

(L  ()  L?  UN  IN)) 

No 

6,  7,  9.  20 

8 

(L  UN)  L?  (-  LN )) 

No 

6.  8,  9,  15.  21 

9 

(L  {LN)  L?  UN  IN)) 

No 

7.  9,  16,  22 

10 

(L  (L-)  M7  ()) 

No 

1,  11,  13,  2  7 

1  1 

(L  (L-)  M7  UN)) 

No 

2.  10.  11,  1 4,  28 

12 

(L  <♦)  M?  UN)) 

No 

2.  12,  29,  64,  66 

13 

(L  (IN  l  )  M7  ()) 

No 

2.  13.  14,  30 

1  4 

(L  {LN  L  )  M?  UN)) 

No 

3,  13,  14,  31 

15 

(L7  (-)  L  {IN)) 

No 

4,  15,  16,  18,  5 7 

16 

(L?  UN  )  l  UN)) 

No 

6,  16.  23,  61 

1  7. 

(L?  (  )  L?  (  )) 

No 

19. 

56 

18 

(L?  (-)  L7  (  IN)) 

No 

1  7, 

18,  20,  23,  56 

1  9 

(L?  (  )  l?  {LN  )) 

No 

19. 

24,  58,  227 

20 

(I.7  (  )  L7  {IN  IN)) 

No 

1  9, 

20,  25.  59 

21 

(l7  (  IN)  L?  (  IN)) 

No 

18, 

21,  22 

22 

(L7  (  LN)  l?  {IN  IN)) 

No 

20, 

22.  23.  26 

23 

(l7  {IN  -)  L7  (  IN)) 

No 

19, 

23,  25,  60 

24 

(l7  {IN  -)  L7  {IN  -)) 

No 

24, 

62 

25 

(l7  {IN  )  L7  {IN  IN)) 

No 

24, 

25.  63 

26 

(L7  UN  IN)  L?  {LN  IN)) 

No 

25. 

26 

27 

(L?  {  l  )  M7  ()) 

No 

4,  28,  32 

28 

(l7  (-1  )  M7  (IN)) 

No 

16, 

27,  28.  33 

29 

U7  (  ♦)  M7  UN)) 

No 

15. 

29.  34,  67 

30 

(L7  (  INI  )  M7  ()) 

No 

5,  31.  35 

31 

(L7  (  LN  l  )  M7  {IN)) 

No 

8,  30.  31 ,  36 

32 

(L?  {LN  L  )  m  {)) 

No 

6,  32.  33 

33 

(L7  {IN  l  )  M7  {IN)) 

No 

16. 

32.  33 

34. 

(L?  {LN  -  *)  ¥?  UN)) 

No 

16. 

34,  69 

36 

(L?  {LN  INI-)  M7  ()) 

No 

7,  35,  36 

36 

(L?  (LN  LN  L- )  M7  {LN)) 

No 

9,  35.  36 

37 

(L?  0  M7 1  (*  A  -)) 

No 

42, 

126,  236 

38 

(L?  ()  M7 1  (♦  A  LN)) 

No 

37, 

38,  43,  47 

39 

(L?  ()  M7 1  (♦  A  IN)) 

No 

39, 

44.  48,  124 

40 

(L?  ()  M7 1  (♦  A  LN  1) 

No 

46, 

49,  237 

41 

(L?  0  M?1  (♦  A  LN  IN)) 

No 

40, 

41,  46,  50 

42 

(L?  0  M?1  {IN  ♦  A  )) 

No 

42. 

153.  238 

43 

(L?  ()  W»71  (LN  *  A  -  IN)) 

No 

42, 

43.  51 

44 

(L?  ()  M71  (LN  *  A  LN)) 

No 

44. 

52.  151 

45 

(l?  ()  M71  (IN  ♦  A  IN  )) 

No 

53,  239 

46 

(L7  ()  M?1  (IN  ♦  A  IN  LN)) 

No 

45. 

46,  54 

4  7 

(L7  (LN)  M7 1  (♦  A  IN)) 

No 

38. 

4  7,  61,  126 

48 

(L?  UN)  M?1  (♦  A  LN)) 

No 

39. 

48.  52.  126 

49 

(L7  (LN)  M7 1  (♦  A-  LN  -)) 

No 

40. 

49.  53,  240 

60 

(L?  (LN)  M?1  (♦  A  LN  LN)) 

No 

41 . 

49.  60,  54 
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1  able  B.l:  Member  ship  Protocol  link  Conditions 


Is  Link 

Number  Link  Condition 

In  Troe?  Successors 

51 

(L?  UN)  M71  UN  ♦  A-  -  IN)) 

No 

43,  51, 

153 

52. 

(L?  UN)  M?  1  UN  ♦  A  IN)) 

No 

44.  52. 

152 

53. 

(L?  UN)  M71  UN  ♦  A  IN  -)) 

No 

46,  53, 

241 

54 

(L?  UN)  M71  UN  ♦  A  IN  IN)) 

No 

46,  53, 

54 

66 

(L?  ()  N?  1  (A  -)) 

No 

3  7,  22 i 

262 

56 

(L?  ()  N71  (A  IN)) 

No 

38,  55, 

66,  60 

67 

(L?  ()  N?  1  (A  IN)) 

No 

39,  57. 

61,  225 

56 

(L?  ()  N71  (A-  IN  -)) 

No 

40,  62. 

253 

59 

(L?  ()  N?1  (A  IN  IN)) 

No 

41.  58. 

59,  63 

60 

(L?  UN)  N71  (A  IN)) 

No 

4  7,  56, 

60,  227 

61 

(L7  UN)  N7 1  (A  IN)) 

No 

48.  67. 

61.  226 

62 

(L?  UN)  N7 1  (A  LN  )) 

No 

49.  68, 

62,  264 

63 

(l7  UN)  N?1  (A  IN  IN)) 

No 

50.  59. 

62,  63 

64 

(M7  ()  L  (♦)) 

No 

1,12,  65,  67 

65 

(M?  ()  L  UN  *)) 

No 

2,  65,  66,  68 

66 

(M7  UN)  L  UN  ♦)) 

No 

3,  65.  66.  71 

6  r 

(M7  ()  l?  (-  ♦)) 

No 

4.  29,  69 

38 

(M?  ()  L?  (  IN  ♦)) 

No 

6,  70.  71 

69 

(M7  ()  L?  UN  -  ♦)) 

No 

6,  34,  69 

7 0 

(M7  (1  L7  UN  IN  ♦)) 

No 

7,  70,  7 

’2 

71. 

(M7  UN)  L?  (  IN  ♦)) 

No 

8,  68,  71.  72 

7? 

(M7  UN)  L?  UN  IN  ♦)) 

No 

9.  70.  72 

7  3 

(M7  (♦)  M7  (♦)) 

No 

64,  74 

7  4 

(M7  (♦)  M?  UN  ♦)) 

No 

12.  65. 

74.  75 

75 

(M7  UN  ♦)  M7  UN  ♦)) 

No 

66,  75 

76 

(M7 1  ()  L  (A*)) 

Yes 

82.  88. 

ioo,  301 

7  7. 

(M7 1  ()  L  (A*  IN)) 

Yes 

76.  77, 

83,  89.  101 

76 

(M71  OKI  A  )) 

No 

10.  84, 

90,  102 

7  9 

(M7 1  Old-  A  IN)) 

No 

78.  70, 

85.  91,  103 

80 

(M7 1  ()  l  (♦  A  )) 

No 

64i  86, 

92,  104 

81 

IM7 1  ()  L  (♦  A  IN)) 

No 

80,  81. 

8  7.  93.  105 

82 

(M7 1  ()  L  UN  AO) 

Yes 

82.  94, 

106,  302 

83 

(M7 1  ()  L  UN  A*  IN)) 

Yes 

82.  83. 

96.  107 

84 

(M7 1  ()  L  UN  l  A  )) 

No 

13.  84, 

96,  108 

86 

(M7 1  ()  L  (LN  l  A  IN)) 

No 

84,  86, 

9  7,  109 

86 

(M71  ()  L  (LN  ♦  A  )) 

No 

65.  86, 

98,  110 

67 

(M7 1  ()  L  UN  *  A  IN)) 

No 

86,  87, 

99.  Ill 

88 

(M7 1  UN)  L  (A*)) 

Yes 

76,  88. 

94.  127.  303 

89 

(M7 1  UN)  L  (A*  LN)) 

Yes 

77.  83, 

89,  95,  128 

90 

( M7  1  UN)  L  (L  A  )) 

No 

11,  78. 

90,  96,  129 

91 

(M?1  UN)  L  (L  A  LN)) 

No 

79.  <U', 

91,  97.  130 

92 

(M7 1  (IN)  L  (♦  A  - )) 

No 

12.  80, 

92.  98.  131 

93 

(M7 1  UN)  L  (♦  A  LN)) 

No 

81,  92, 

93.  99.  132 

94 

(M7 1  (LN)  L  UN  A*)) 

Yes 

82,  94. 

133,  304 

95 

(M?1  (LN)  l  UN  A*  IN)) 

Yes 

83,  94, 

95,  134 

96 

(M71  UN)  L  UN  l  A  )) 

No 

14,  84, 

96,  136 

9  7 

(M7 1  UN)  L  UN  L  A  IN)) 

No 

85,  96, 

97.  136 

98 

(M7 1  UN)  L  UN  *  A  )) 

No 

66.  86, 

98,  137 

99 

(M7 1  UN)  L  (IN  ♦  A-  IN)) 

No 

87.  98, 

99,  138 

100 

(M7 1  ()  L?  (-  AO) 

Yes 

112,  127,  310 
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I  able  P.1:  Member  shi  p  Protocol  link.  Conditions 


Is  Link 

Number  link  Condition  In  Tre*}?  Successors 


101 

(M71 

0  L?  (-  A*  LN)) 

Yes 

100,  101.  113,  128 

102. 

(M71 

0  L?  (  L  A  )) 

No 

27.  114.  129 

103 

(M71 

0  L7  (-  l-  A  IN)) 

No 

102,  103,  115.  130 

104 

(M71 

0  L?  (  ♦  A  )) 

No 

67.  116,  131 

106 

(M7 1 

()  L7  (  ♦  A  IN)) 

No 

104,  106.  11  7,  132 

106 

(M7  1 

()  L?  (  IN  AO) 

Yes 

1  18  133.  31  1 

10/. 

(M7 1 

()  L?  (  LN  k*  LN)) 

Yes 

106,  107,  119,  134 

108 

(M7  1 

0  L?  (-  IN  L-  A-)) 

No 

30,  120,  135 

109 

(M7  1 

()  L?  (  LN  L-  A  IN)) 

No 

108,  109,  121,  136 

1  10 

(M7  1 

0  L?  (  LN  *  A  )) 

No 

68,  122,  137 

1  1  1 

(M7  1 

()  L?  (  IN  ♦  A  IN)) 

No 

110.  Ill,  123.  138 

1  12 

( M7  1 

()  l?  (LN  A*)) 

Yes 

1  12,  139,  313 

1  13 

( M7  1 

()  L?  (IN  A*  IN)) 

Ves 

112,  113,  140 

1  1  4 

(M7  1 

0  l7  (LN  L-  A  )) 

No 

32,  114.  141 

1  16 

(M7  1 

()  L7  (IN  -  l~  A  IN)) 

No 

114,  115,  142 

1  16 

( M7  1 

()  L7  (IN  ♦  A  )) 

No 

69.  116.1  43 

1  1  7 

(M7  1 

()  L7  (LN  *  A  IN)) 

No 

116,  117,  144 

1  18 

(M7  1 

()  l7  UN  -  LN  A*)) 

Yos 

1  18,  145,  314 

1  19 

(M7  1 

()  L?  (IN  LN  A*  LN)) 

Yes 

118,  119,  146 

120 

(M7  1 

()  L7  (IN  LN  1  A  )) 

No 

36,  120,  147 

121 

(M7  1 

()  L7  (IN  LN  l  A  IN)) 

No 

120,  121,  148 

122 

(M7  1 

0  L7  (IN  IN  ♦  A  )1 

No 

70,  122,  149 

123 

( M7  1 

()  L7  (IN  IN  *  A  IN)) 

No 

122,  123.  150 

124 

(M7 1 

(*  A  )  L7  ()) 

No 

126,  151,  180 

126 

(M7  1 

(♦A  )  L7  (IN)) 

No 

124,  126,  162,  181 

126 

( M7  1 

(♦A  )  L7  UN)) 

No 

3  7,  1  26.  1  53.  200 

1  27 

(M7  1 

(IN)  L7  (  A*)) 

Yes 

100,  127,  139,  316 

128 

(M7  1 

(IN)  L7  (-  A*  IN)) 

Yes 

101.  127.  128,  140 

129 

(M7  1 

(IN)  L7  (  l  A  )) 

No 

28.  102,  129,  141 

130 

(M7  1 

(IN)  L7  (  l  A  LN)) 

No 

103,  129,  130,  142 

13’ 

(M7  1 

(IN)  L7  (  ♦  A  )) 

No 

29.  104,  131,  143 

132 

(M7  1 

(LN)  L7  (  ♦  A  IN)) 

No 

105,  131.  132,  144 

133 

(M7  1 

(IN)  l7  (  IN  A*)) 

Yes 

106,  133,  145,  31  7 

134 

(M7  1 

(IN)  L7  (  IN  A ♦  IN)) 

Yes 

107,  133,  134,  146 

136 

(M7  1 

(IN)  L7  (  IN  (  A  )) 

No 

31,  108,  135,  147 

136 

(M7  1 

(IN)  L7  (  IN  L-  A  IN)) 

No 

109.  135,  136,  148 

137 

(M7  1 

(IN)  L7  (-  IN  *  A  )) 

No 

71,  1  10.  137,  149 

138 

(M7 1 

(IN)  L7  (  IN  ♦  A  IN)) 

No 

111,  13  7,  138,  1 50 

139 

(M7  1 

UN)  L7  UN  A*)) 

Yes 

1  12,  139,  319 

1  40 

(M7 1 

(IN)  L7  UN  A*  IN)) 

Yes 

1  13,  139,  140 

141 

(M7  1 

UN)  L7  (IN  t  A  )) 

No 

33.  114.  141 

142 

(M7  1 

(IN)  L7  (IN  l  A  IN)) 

No 

115.  141,  142 

143 

(M'  1 

(IN'1  L7  (IN  -  *  A  )) 

No 

34.  1  16.  1  43 

1  44 

( M7  1 

(IN)  L?  (IN  *  A  IN)) 

No 

1  1  7.  143,  144 

145. 

(M7  1 

(LN)  L7  (IN  IN  A*  i) 

Yes 

116,  146,  320 

1  46 

(M71 

(IN)  L7  UN  IN  A*  IN)) 

Yes 

1  19.  145.  146 

147 

(M7  1 

(IN)  L7  (IN  IN  l  A  )) 

No 

36,  120,  147 

148 

(M7  1 

(LN)  L7  UN  -  IN  I  A  IN)) 

No 

121,  147,  148 

149 

(M7  1 

(IN)  L7  UN  -  IN  *  A-)) 

No 

72.  122,  149 

160 

( M7 1 

UN)  L?  UN  IN  *  A  LN)) 

No 

123,  149.  150 
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Is  Link 

Number  l  ink  Condition  In  Tree?  Successors 


161 

162. 

1  63. 
164 

1  66 

(M71 
(M7  1 
(M7  1 
(M7  1 
(M7  1 

UN  ♦  A  )  L?  ()) 

UN  *  k  )  L?  UN)) 

UN  ♦  A  -)  L?  (.'A/)) 

(♦)  M7  (A*)) 

(♦)  M7  (A+  IN)) 

No 

No 

No 

Yes 

Yes 

161.  162,  18? 

161,  162,  183 

42.  163,  201 

76.  168,  162,  367 

77,  164,  166,  169,  163 

166 

(M7 1 

(♦)  M7  (♦  A  )) 

No 

73.  80,  160,  164 

167 

(M7  1 

(♦)  M?  (♦  A  LN)) 

No 

81,  166,  167,  Ibl.  166 

168 

(M7 1 

(♦)  M7  UN  A*)) 

Yes 

82,  158,  166,  358 

1  69 

(M7  1 

(*)  M?  (LN  A*  IN)) 

Yes 

83,  158,  169,  16/ 

160 

(M7  1 

(♦)  M7  (LN  ♦  A  )) 

No 

74,  86.  160,  168 

16  1 

(M7  1 

(♦)  ¥■>  (IN  *  A  1  V)) 

No 

8  7,  160,  161,  169 

1  02 

(M7  1 

UN  *)  M7  (A» )) 

Yes 

88,  162,  166,  369 

163. 

(M7 1 

UN  ♦)  M7  (A*  IN)) 

Yes 

89,  162,  163,  167 

164 

(M7  1 

(IN  ♦)  M?  (♦  A  )) 

No 

74,  92.  164,  168 

1  66 

( M?  1 

UN  ♦)  M7  (♦  A  IN)) 

No 

93.  1 64,  166,  169 

166 

(M7  1 

UN  *)  M7  (IN  A*)) 

Yes 

94.  16(i,  360 

10/ 

(M7 1 

(LN  *)  M7  UN  A ♦  IN)) 

Yes 

95,  166,  167 

168 

(M7  1 

(IN  ♦)  M?  (IN  *  A  )) 

No 

75.  98.  168 

169 

(M7 1 

UN  ♦)  M?  UN  ♦  A  IN)) 

No 

99.  168,  169 

1  70 

(M7  1 

(♦  A  )  M7 1  (♦  A  )) 

1  56,  1  7? 

1  7  1 

(M7  1 

(♦  A  )  M7 1  A  IN)) 

No 

1  6  7,  1  70.  1  71  .  1  73,  1  76 

1  72 

(M7  1 

(♦  A  )  M7 1  UN  *  A  )) 

No 

160,  164,  1  72.  1  77 

1  73 

(M7  1 

(♦  A  )  M7 1  (IN  *  A  IN)) 

No 

161 ,  1  72,  1  73,  1  78 

1  74 

( M7  1 

(♦  A  IN)  M7 1  (♦  A  IN)) 

No 

171,  174,  176 

1  7  6 

(M7  1 

(*  A  IN)  M7 1  (IN  *  A 

IN)) 

No 

1  73,  1  76,  1  76,  1  79 _ 

1  76 

(M7  1 

(LN  *  A  )  M7 1  (♦  A  IN)) 

No 

165,  1  72.  1  76.  1  78 

177 

(M7  1 

(IN  *  A  )  M7 1  UN  ♦  A 

)) 

No 

168,  177 

1  78 

(M7  1 

(IN  *  A  )  M7 1  UN  *  A 

IN)) 

No 

169,  1  77.  1  78 

1  79 

(M7  1 

UN  ♦  A  IN)  M7 1  (IN  ♦ 

A  IN)) 

No 

1  78.  1  79 

180 

( M7  1 

(♦)  N  (A  )) 

No 

78,  166,  182,  210,  222 

181 

(M7  1 

(♦)  N  (A  IN)) 

No 

79,  167.  180,  181,  183.  2  1  1 

18? 

(M7  1 

(IN  ♦)  N  (A  )) 

No 

90.  164,  182,  212,  223 

1  83 

(M7  1 

(IN  *)  N  (A  IN)) 

No 

91.  166,  182,  183,  213 

184 

(M7 1 

( ♦  A  )  N!  ()) 

Yes 

164,  186,  188 

186 

(M7  1 

!♦  A  )  N!  (IN)) 

Yes 

156,  164,  185,  189 

1  86 

(M7 1 

(♦A  IN)  W  O) 

Yes 

184,  186,  18  7.  1  90 

187. 

(M7 1 

(♦  A  IN)  N!  (IN)) 

Yes 

185,  186.  187,  191 

188 

(M7 1 

UN  *  A  )  N!  ()) 

Yes 

162,  188,  189 

1  89 

(M7  1 

(IN  *  A  )  N!  (IN)) 

Yes 

163,  188,  189 

190 

(M7  1 

(IN  *  A  IN)  N!  ()) 

Yes 

188,  190,  191 

191 

(M7  1 

(IN  *  A  'IN)  Nl  UN)) 

Yes 

1  89,  190,  191 

19? 

(M7 1 

(♦  A  )  N7  ()) 

No 

180,  193,  196,  214 

193 

(M7  1 

(♦  A-)  N7  (IN)) 

No 

181,  192,  193,  19  7,  215 

194 

(M71 

(♦  A  LN)  N7  ()) 

No 

192,  194,  196.  198,  216 

1  95 

(M7  1 

(♦  A  IN)  N?  (LN)) 

No 

193,  194,  195,  199,  217 

1  06 

( M7  1 

(IN  *  A  )  N?  ()) 

No 

182,  196,  197,  218 

IP  7 

(M7 1 

(IN  *  A  )  N7  (IN)) 

No 

183,  196,  197,  219 

198 

(M71 

(IN  ♦  A  LN)  H?  0) 

No 

196,  198,  199,  220 

199 

(I471 

(IN  ♦  A  IN)  N7  (IN)) 

No 

197,  198,  199,  221 

200 

(M7 1 

(♦  A  )  N?  1  (A  LN)) 

No 

171,  181,  200,  201,  236 
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fable  8.1:  Membership  Protocol  Link  Conditions 


Is  LinA 

Numt»‘r  link  Condition  In  Tree ?  Successors 


pot 

(M71 

UN  ♦  A  )  N71  (A-  thl)) 

No 

1  26. 

183, 

201 . 

238 

POP 

(M7 1 

0  S  (L*  A  1) 

Yes 

204, 

206, 

265 

?  03 

(wri 

()  S  (l  ♦  A-  IN)) 

Yos 

202, 

203, 

206, 

20/ 

204 

(M7 1 

()  S  UN  L ♦  A  )) 

Yes 

204, 

208, 

26  7 

P0t> 

(M71 

()  S  (IN  i*  A  IN)) 

Yes 

204, 

206, 

209 

P  06 

(M7 1 

UN)  S  (l  ♦  A-)) 

Yes 

202, 

200. 

208, 

266 

20f 

(M7 1 

UN)  S  (l  ♦  A  IN)) 

Yes 

203, 

206, 

20/. 

209 

208 

(M7  1 

UN)  S  UN  l ♦  A- )) 

Yes 

204, 

208, 

268 

209 

(M?1 

UN)  S  UN  l  ♦  A  IN)) 

Yes 

205, 

208, 

209 

210 

(M?  1 

(*)  X  (A  )) 

No 

180, 

POP. 

212, 

56/ 

21  1 

(M7 1 

(♦)  X  (A  IN)) 

No 

181. 

203, 

210. 

211, 

213 

212 

(M7 1 

UN  ♦)  X  (A  )) 

No 

182, 

206, 

212, 

558 

213 

(M7  1 

UN  ♦)  X  (A  IN)) 

No 

183, 

202, 

212, 

213 

214 

(M7  1 

(♦A  )  X7  ()) 

No 

192, 

210. 

218 

215 

(M7 1 

(♦  A  )  X?  UN)) 

No 

193, 

21  1. 

214, 

216. 

219 

21f> 

(M7  1 

(♦  A-  IN)  X7  ()) 

No 

184, 

186, 

194, 

220 

21  2 

(M7  1 

(♦  A  LN)  X?  UN)) 

No 

1  85. 

182, 

196, 

216, 

21  /,  221 

2  1  8 

(M7  1 

UN  ♦  A- )  X?  ()) 

No 

196, 

21  2. 

218 

219 

(M7 1 

UN  ♦  A  )  X?  UN)) 

No 

19/. 

213, 

218, 

219 

220 

(M7  1 

UN  ♦  A  IN)  X7  ()) 

No 

188, 

1  90, 

198, 

220 

221 

(M7 1 

(IN  ♦  A  IN)  X7  UN)) 

No 

189, 

191, 

1  99, 

220. 

221 

222 

(N  () 

M7  (♦)) 

No 

10.  23.  223,  552 

223 

(N  () 

M7  UN  ♦)) 

No 

11.  24,  223,  658 

224 

(n  n 

N  ()) 

No 

P2P. 

659 

PPt) 

(N7 1 

(A  )  l7  <)) 

No 

124, 

226, 

242 

PPC) 

(N7  1 

(A  )  L7  UN)) 

No 

125, 

225, 

226, 

243 

222 

(N7t 

(A  )  L?  UN)) 

No 

56.  1 

26.  227.  263 

22b 

(N7 1 

()  M7  (A*)) 

Yes 

1  54, 

232, 

265 

229 

(N7  1 

(1  M7  (A*  LN)) 

Yos 

1  65. 

228, 

229. 

233 

230 

(N?1 

()  M7  (♦  A  )) 

No 

1  56, 

P2P. 

234 

231 

(N7  1 

()  M7  (♦  A  IN)) 

No 

152, 

230, 

231, 

236 

23? 

(N?1 

0  M?  UN  A*)) 

Yes 

1  68. 

232, 

266 

233 

(N71 

()  M7  UN  A*  IN)) 

Yes 

169, 

232. 

233 

234 

(N7  1 

( )  M?  U  N  *  A  )) 

No 

160, 

223, 

234 

235 

(N7 1 

()  m  UN  *  A  IN)) 

No 

161, 

234. 

236 

230 

fa7  i 

(A  )  M7 1  (♦  A  )) 

No 

1  20, 

180, 

230, 

238 

23  2 

(N7 1 

(A  )  M7 1  (♦  A-  / N1 ) 

No 

121. 

231. 

236. 

23/. 

239 

238 

(N?  1 

IA  )  M7 1  UN  ♦  A  )) 

No 

1  22. 

182. 

234. 

238 

239 

(N71 

(A  )  M7 1  (IN  *  A  IN)) 

No 

1  23. 

23 5, 

238. 

239 

240 

(N7  1 

(A  IN)  M7 1  (♦  A  IN)) 

No 

124, 

200, 

23/ , 

240, 

241 

24  1 

(N7 1 

(A  IN)  M7  1  UN  ♦  A  IN) ) 

No 

1  25, 

201, 

239, 

241 

242 

(N7 1 

0  N  (A  )) 

No 

180, 

224. 

230. 

265 

243 

(N7  1 

()  N  (A  IN)) 

No 

181, 

231 . 

242. 

243, 

266 

244 

(N7 1 

(A  )  Nl  ()) 

Yes 

1  84. 

228. 

245 

245 

(N?  1 

(A  )  Nl  UN)) 

Yes 

185, 

229, 

244, 

246 

246 

(N7 1 

(A  iN)  Nl  (1) 

Yes 

186, 

244. 

246. 

24/ 

242 

(N7 1 

(A  IN)  N'  UN)) 

Yes 

182, 

246, 

246, 

24/ 

248 

(N?1 

(A  )  N?  ()) 

\'r* 

192, 

242, 

249. 

25/ 

249 

(N7 1 

(A  )  N7  UN)) 

No 

193, 

243, 

248. 

249, 

268 

250 

(N7 1 

(A  IN)  N7  ()) 

No 

194. 

248, 

250. 

261, 

269 
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Is  link 

Number  link  Condition 

In  T me?  Successors 

251 

(N?1  (A  IN)  N?  (IN)) 

No 

195. 

249, 

250, 

251, 

260 

262 

(N?1  (A  )  N?1  (A  )) 

No 

236, 

242 

263. 

(N?1  (A  )  N7  1  (A  IN)) 

No 

200, 

23  7, 

243. 

262, 

253 

2  54 

(N?  1  (A  IN)  N?1  (A  IN)) 

No 

240. 

253, 

254 

255 

(N71  ()  X  (A-)) 

No 

2  <0, 

242, 

559 

26  6 

(N7  1  0  X  (A  IN)) 

No 

21  1. 

243. 

265, 

256 

257 

( N?  1  (A  )  X?  ()) 

No 

214. 

248, 

265 

2b  8 

(N?1  (A  )  X7  (IN)) 

No 

215. 

249, 

256, 

257, 

268 

2  b  9 

(N?1  (A  IN)  X?  ()) 

No 

216, 

244, 

246, 

250 

260 

(N?1  (A  IN)  X7  (IN)) 

No 

21  /, 

245, 

247, 

251  , 

259,  260 

261 

(S  ()  M  ()) 

Yes 

262. 

263. 

285. 

293 

262 

IS  ()  M  (IN)) 

Yes 

261 . 

262. 

264. 

286, 

294 

263 

(S  (IN)  M  ()) 

Yes 

2b  1. 

263. 

264 

289. 

29  7 

264 

(S  (IN)  M  (LN)) 

Yes 

262. 

263, 

264. 

290. 

298 

265 

(S  (l»)  M7  ()) 

Yes 

261  . 

266, 

26  7 

266 

(S  (l  ♦)  M7  (IN)) 

Yes 

262, 

266, 

266, 

268 

26/ 

(S  UN  l  ♦)  M’  ()) 

Yes 

263, 

26/, 

268 

268 

(S  (IN  l  ♦)  M?  (/A/)) 

Yes 

264, 

267, 

268 

269 

(S  ()  N!  (  )) 

Yes 

244, 

2/1. 

2/3 

2/0 

(S  ()  H>  (  IN)) 

Yes 

269, 

2/0. 

2  72. 

274 

2/1 

(S  0  Nt  UN  )) 

Yes 

245, 

2/1. 

275 

2/2 

(S  0  N!  UN  IN)) 

Yes 

2/1 , 

272, 

2/6 

2/3 

(S  (IN)  N!  (  )) 

Yes 

246, 

269. 

273. 

2  76 

2/4 

(S  UN)  N*  (  IN)) 

Yes 

2/0, 

273, 

2/4, 

2  76 

2  lb 

(S  UN)  N!  UN  )) 

Yes 

24/. 

2/1. 

2/5 

2/6 

(S  UN)  N<  UN  IN)) 

Yes 

2/2, 

2  7  b. 

2/6 

211 

(S  ()  N7  (  )) 

No 

248, 

2/9, 

281, 

293 

2/8 

(S  ()  N7  (  IN)) 

No 

2//, 

278, 

280. 

282, 

294 

2/9 

(S  0  N7  UN  )) 

No 

249. 

279. 

283, 

296 

280 

(S  ()  N7  UN  IN)) 

No 

2/9, 

280. 

284. 

296 

28  1 

(S  UN)  N7  (  )) 

No 

250. 

27  7. 

281 . 

283, 

297 

282 

(S  UN)  N7  (  IN)) 

No 

2  i  A. 

281 , 

282, 

284. 

298 

283 

(S  UN)  N7  (IN  )) 

No 

251, 

2/9. 

283. 

299 

284 

(S  UN)  N7  (IN  IN)) 

No 

280. 

283, 

284, 

300 

286 

(S  0  S  (♦)) 

Yes 

261, 

28/, 

289 

286 

(S  ()  S  (♦  IN)) 

Yes 

285, 

286. 

288. 

290 

28/ 

(SOS  (IN  ♦)) 

Yes 

263. 

28  7, 

291 

288 

(S  ()  S  (IN  *  IN)) 

Yes 

28/. 

288, 

292 

280 

(S  UN)  S  (♦)) 

Yes 

262. 

285, 

289, 

291 

290 

(S  (IN)  S  (♦  IN)) 

Yes 

286, 

289. 

290, 

292 

291 

(S  (IN)  S  UN  *)) 

Yes 

264. 

28  7. 

291 

292 

(S  (IN)  S  (IN  ♦  IN)) 

Yes 

288. 

291 . 

292 

293 

(S  ()  X7  (  )) 

No 

25/, 

277. 

297 

294 

(S  ()  X7  (  IN)) 

No 

2/8. 

293, 

294, 

298 

296 

(S  ()  X7  (IN  )) 

No 

258, 

279. 

299 

296 

(S  ()  X7  UN  IN)) 

No 

280, 

295, 

296. 

300 

29/ 

(S  (IN)  X7  (  )) 

No 

269, 

269, 

2  73, 

231, 

297 

298 

(S  UN)  X?  (  LN)) 

No 

270. 

274. 

282, 

297, 

298 

299 

(S  UN)  X7  (IN  -)) 

No 

260, 

271. 

2  75. 

283, 

299 

300 

(S  (IN)  X7  (IN  IN)) 

No 

272. 

276, 

284, 

299. 

300 
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fatlo  8.1:  Membership  Protocol  Link  Conditions 


Is  Link 

Number 

link  Condition 

In  Iroe ?  Successors 

301 

(S? 

(A*)  L  ()) 

Yes 

302. 

30ft, 

310, 

333 

30?. 

(S’ 

(A*)  L  UN)) 

Yes 

301, 

302, 

306. 

311, 

336 

303 

(S? 

(A*  IN)  L  ()) 

Yus 

301 . 

303, 

304, 

30/, 

316 

304 

(S'* 

(A*  LN)  l  (IN) ) 

Yes 

302, 

303, 

304, 

308, 

31  / 

30b 

(S? 

(IN  A*)  L  ()) 

Yes 

306, 

306, 

322. 

34  b 

306 

(S? 

UN  A*)  L  UN)) 

Yes 

30ft, 

306, 

323, 

348 

30/ 

(S’ 

UN  A*  LN)  L  ()) 

Yes 

306. 

30/, 

308, 

328 

308 

(S’ 

UN  A ♦  IN)  l  UN)) 

Yes 

306, 

30/. 

308, 

329 

309 

(S? 

( A ♦)  L?  ()) 

Yes 

312, 

321, 

333 

310 

(S’ 

(A*)  L?  (  )1 

Yes 

309, 

313, 

322, 

334 

31  1 

(S’ 

(A*  1  L?  1  IN)) 

Yes 

310, 

311, 

314, 

323, 

335 

312 

(S’ 

(A*)  L?  UN)) 

Yes 

309, 

312, 

324, 

336 

313 

(S'* 

(A*  1  l?  (IN  )) 

Yes 

312, 

313, 

32ft, 

33/ 

314 

(S? 

(A*)  L?  UN  IN)) 

Yes 

313, 

314, 

32b, 

338 

31  ft. 

(S'* 

(A*  IN)  L?  ()) 

Yes 

309, 

316, 

318, 

32/ 

316 

(S’ 

(A*  IN)  l’  (  )) 

Yes 

310, 

3  1  6. 

316, 

319, 

328 

3  1  /. 

(S’ 

(A*  LN)  l’  (  LN)) 

Yes 

311, 

316, 

31  /. 

320, 

329 

318 

(S’ 

(A*  IN)  L?  (IN)) 

Yes 

312, 

316, 

318, 

330 

319 

(S’ 

(A*  LN)  L?  (LN  )) 

Yes 

313, 

318, 

319, 

331 

320 

(S’ 

(A*  IN)  L?  (IN  IN)) 

Yes 

314, 

319, 

320, 

332 

321 

(S’ 

UN  A*)  l’  ()) 

Yes 

321 , 

324, 

34ft 

322 

(S’ 

UN  A*)  L’  (  )) 

Yes 

321. 

322, 

32ft, 

346 

323 

(S’ 

UN  A ♦)  L’  (  IN)) 

Yes 

322, 

323. 

326, 

34? 

324 

(S’ 

UN  A*)  L’  (LN)) 

Yes 

321, 

324. 

348 

32ft 

(S’ 

(IN  A*)  L’  (IN  )) 

Yes 

324, 

326, 

349 

326 

(S’ 

UN  A*)  l’  (IN  IN)) 

Yes 

326, 

326. 

3ft0 

32/ 

(S’ 

UN  A ♦  LN)  L’  ()) 

Ves 

321, 

32/, 

330 

328 

(S’ 

UN  A*  IN)  L?  (  )) 

Vos 

322, 

32/. 

328, 

331 

329 

(S’ 

(IN  A*  IN)  L’  (  IN)) 

y«s 

323, 

328, 

329, 

332 

330 

(S’ 

UN  A*  IN)  l?  (IN)) 

y«s 

324, 

32/. 

330 

331 

(S’ 

(LN  A*  IN)  L’  (IN  )) 

yes 

3?ft, 

330, 

331 

332 

(S’ 

(LN  A*  LN)  L?  (IN  IN)) 

yes 

326, 

331, 

332 

333 

(S’ 

0  M  (A*)) 

yes 

261, 

339, 

34ft, 

461, 

609 

334. 

(S’ 

0  M  (A*  )) 

yes 

333, 

340, 

346, 

462, 

510 

33ft 

(S’ 

()  M  (A*  IN)) 

yes 

334, 

33b, 

341, 

34/, 

463, 

61  1 

336 

(S’ 

()  M  (A*  LN)) 

yes 

333, 

336, 

342. 

348, 

484, 

612 

337 

(S’ 

()  M  (A*  IN  )) 

yes 

336. 

343, 

349, 

465, 

513 

338 

tS’ 

()  M  (A*  IN  IN)) 

Yes 

33  /, 

338, 

344, 

350. 

406, 

514 

339 

(S’ 

()  M  (LN  A*)) 

Yes 

262, 

339, 

351. 

467, 

515 

340 

(S’ 

()  M  (IN  A*  -)) 

Yes 

339, 

340, 

352, 

468, 

516 

341 

(S’ 

0  M  (LN  A*  IN)) 

Yes 

340, 

341, 

353, 

469, 

51  7 

34? 

(S’ 

O  M  (LN  A ♦  IN)) 

Yes 

339, 

342, 

354, 

4/0, 

518 

343 

(S’ 

()  M  (LN  A *  IN  )) 

Yes 

342, 

343, 

35ft. 

471, 

519 

344 

(S’ 

()  M  (LN  A*  LN  IN)) 

Yes 

343. 

344. 

356, 

4/2, 

620 

34ft 

(S’ 

UN)  M  ( A ♦ ) ) 

Yes 

263, 

333, 

34ft, 

351, 

485, 

533 

346 

(S’ 

UN)  M  (A*  )) 

Yes 

334, 

346, 

346, 

352, 

486, 

534 

347 

(S’ 

UN)  M  ( A ♦  -  IN)) 

Yes 

336, 

346, 

34  7, 

353, 

487, 

53  ft 

348 

(S’ 

(LN)  M  (A*  LN)) 

Yes 

336, 

345, 

348, 

354, 

488, 

536 

349 

(S’ 

UN)  M  (A*  IN  -)) 

Yes 

33  7, 

348, 

349, 

356, 

489. 

537 

350 

(S’ 

UN)  M  (A*  LN  LN)) 

Yes 

338, 

349, 

350, 

356, 

490, 

638 
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Is  Link 

Number 

L 

ink  Condition 

In  tree?  Successor  s 

361 

(S7 

UN) 

M  UN  A*)) 

yes 

264. 

330, 

351, 

491, 

539 

362 

(S7 

UN) 

M  UN  A»  )) 

Ves 

340, 

361, 

352, 

492, 

540 

363 

(S7 

UN) 

M  UN  A«  IN)) 

yes 

34  1, 

362. 

363, 

493, 

641 

354 

(S7 

UN) 

M  UN  A*  LN)) 

Yes 

342, 

361 , 

354, 

494. 

542 

366 

(S? 

UN) 

M  UN  A*  IN  )) 

Yes 

343, 

364 

366, 

496, 

543 

366 

(s7 

UN) 

M  UN  A*  IN  IN)) 

Yes 

344, 

355, 

356, 

496, 

544 

36/ 

(S? 

(A*  < 

0  M7 

0) 

Yes 

301, 

368, 

361 

358 

(s7 

(A*  « 

0  M7 

UN)) 

Yes 

302, 

357. 

358, 

362 

359 

(S7 

(A*  LN  *) 

M?  ()) 

Yes 

303, 

360 

363 

360 

(S? 

(A*  / 

N  *) 

M?  {IN)) 

Yes 

304, 

359, 

360, 

364 

361 

(S7 

UN  / 

\*  ♦) 

M?  ()) 

Yes 

306, 

36  1 . 

362 

362 

(S7 

UN  l 

V+  ♦ ) 

M7  UN)) 

Yes 

306, 

361 , 

362 

363 

(S'* 

UN  l 

\*  IN 

*)  M7  0) 

yes 

30/. 

363, 

364 

364. 

(S7 

UN  1 

\*  LN 

♦  )  M7  UN)) 

Yes 

308, 

363, 

364 

366 

(S? 

0  N! 

(-  A* 

)) 

Yes 

269. 

37  7. 

389 

366 

(S7 

()  Ml 

(- 

)) 

Yes 

366, 

3/8, 

390 

36/ 

(S'* 

0  N! 

(  A* 

IN)) 

Yes 

366, 

36/, 

3/9, 

391 

368 

(S7 

0  N! 

(-  A* 

IN)) 

Yes 

366, 

368, 

380, 

392 

369 

(S? 

0  N! 

(  A* 

LN  )) 

Yes 

368, 

381 , 

393 

3/0 

(S7 

0  N! 

(  A* 

IN  IN)) 

Yes 

369, 

3/0, 

302, 

394 

3/1 

(S7 

0  N! 

(  IN 

A*)) 

Yes 

2/0, 

383, 

395 

3/2 

(S'* 

0  Ml 

(  IN 

A»  )) 

Yes 

3/1 , 

384, 

396 

3/3 

(S7 

0  N! 

(  IN 

A*  IN)) 

Yes 

3/2, 

373. 

386. 

39/ 

3/4 

(S7 

0  Ml 

(  IN 

A*  IN)) 

Yes 

3/1 . 

3/4, 

386. 

398 

3/6 

(S’ 

l)  N! 

(  IN 

A*  IN  )) 

Yes 

3/4, 

38/. 

399 

3/6 

(S'’ 

()  N* 

(  IN 

A*  IN  IN)) 

Yes 

3/6, 

3/6. 

388, 

400 

3// 

(S'’ 

0  Ml 

UN 

A*)) 

Yes 

2/1. 

377. 

401 

3/8 

(S'’ 

0  N! 

UN 

A*  )) 

Yes 

3  7  7, 

3/8, 

402 

3/9 

(S'’ 

0  N! 

UN 

A*  IN)) 

Yes 

3/8, 

3/9. 

403 

380 

(S'’ 

0  N! 

UN 

A*  LN)) 

Yes 

37  7, 

380, 

404 

381 

(S'’ 

()  N! 

UN 

A*  LN  )) 

Yes 

380, 

381 . 

405 

382 

(S? 

0  N! 

UN 

A*  LN  IN)) 

Yes 

381 , 

382, 

406 

383 

(S7 

0  N! 

UN 

IN  A*)) 

Yes 

2/2, 

383, 

40/ 

384 

(S7 

0  N! 

UN 

IN  A*  )) 

Yes 

383, 

384. 

408 

386 

(S7 

0  N! 

UN 

LN  A*  LN)) 

Yes 

384, 

385, 

409 

386 

(S? 

0  Ml 

UN 

IN  A*  IN)) 

Yes 

383, 

386, 

410 

38/ 

(S? 

0  N! 

UN 

IN  A*  IN  )) 

Yes 

386, 

38/, 

41  1 

388. 

(S7 

0  N! 

UN 

LN  A*  IN  IN)) 

Yes 

38/, 

388, 

412 

389 

(S7 

UN) 

N!  ( 

A*)) 

Yes 

2/3, 

365, 

389, 

401 

390 

(S7 

UN) 

N!  ( 

A*  )) 

Yes 

366, 

389, 

390, 

402 

391 

(S? 

UN) 

N!  ( 

A^  LN)) 

Yes 

36/. 

390, 

391, 

403 

392. 

(S7 

UN) 

N!  ( 

A»  LN)) 

>es 

368, 

389, 

392, 

404 

393 

(S7 

UN) 

Ml  ( 

A*  IN  )) 

Yes 

369, 

392, 

393, 

405 

394 

(S7 

(LN) 

Ml  ( 

A*  IN  IN)) 

Ves 

3/0, 

393, 

394, 

406 

395 

(S? 

UN) 

Ml  ( 

LN  A*)) 

yes 

27  4^ 

371, 

395, 

407 

396 

(S7 

UN) 

N!  ( 

LN  A*  )) 

Yes 

372, 

396, 

396, 

408 

39/ 

(S? 

UN) 

N!  ( 

IN  A*  IN)) 

yes 

3/3, 

396, 

397, 

409 

398 

(S? 

UN) 

Ml  ( 

LN  A*  LN)) 

yes 

374, 

395, 

398, 

410 

399 

(S7 

UN) 

Nt  (- 

LN  A*  LN  -)) 

Yes 

375, 

398, 

399, 

411 

400 

(S? 

UN) 

N!  ( 

LN  A*  LN  IN)) 

Yes 

3/6, 

399, 

400, 

412 
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Is  Link 

Number  Link  Condition 

In  Tree 7  Successors 

401 . 

(S?  UN)  N!  (IN  -  A*)) 

Yes 

27  b, 

377. 

401 

40?. 

(S?  UN)  N!  UN  A*  )) 

Yes 

376, 

401,  402 

403. 

(S?  UN)  N!  UN  -  A*  -  IN)) 

Ves 

370, 

402, 

403 

404 

(S?  UN)  Nf  UN  A t  LN)) 

Yes 

360. 

401, 

404 

40b 

(S?  UN)  N!  (IN  -  A*  LN  )) 

Yes 

381. 

404, 

405 

406 

(S?  UN)  N!  (LN  A t  LN  -  IN)) 

Yes 

382. 

405, 

406 

40  7. 

(S?  (LN)  Nl  (LN  LN  A*)) 

Yes 

276, 

383, 

407 

406 

(S?  (LN)  N!  (LN  -  IN  A*  -)) 

Yes 

384, 

407, 

408 

409 

(S'7  (LN)  Nl  (LN  LN  A*  LN)) 

Yes 

385, 

408, 

409 

410 

(S?  (LN)  N!  UN  LN  A*  IN)) 

Yes 

386, 

407, 

410 

411 

(S7  (IN)  N!  (IN  IN  A*  IN  )) 

Yes 

387, 

410. 

41  1 

412. 

(S?  (LN)  Nl  UN  IN  A*  IN  IN)) 

Yes 

388, 

41  1, 

412 

413 

( S ?  0  N’  (-  A*)) 

No 

277, 

425, 

437, 

509 

414 

(S?  ()  N?  (  A*  )) 

No 

413. 

426, 

438, 

510 

415 

(S?  ()  N7  (-  A*  -  LN)) 

No 

414, 

415L 

427. 

439, 

61  1 

416 

(S?  0  N?  (  A*  LN)) 

No 

413, 

416, 

428, 

440. 

512 

41  ! 

(S?  ()  N7  (  A*  LN  )) 

No 

416, 

429, 

441 . 

513 

416 

(S?  ()  N?  (-  A*  LN  -  IN)) 

No 

41  7. 

418, 

430, 

442. 

514 

419 

(S?  ()  N7  (  LN  A ♦)) 

No 

278, 

431, 

443, 

515 

4?0 

(S'7  0  N7  (  IN  A t  )) 

No 

419, 

432. 

444, 

516 

421 

(S'7  ()  N7  (  IN  A*  IN)) 

No 

420, 

421. 

433, 

445, 

51  7 

422. 

(S?  0  N7  (  LN  At  IN)) 

No 

419, 

422, 

434, 

446, 

518 

423 

(S'7  ()  N7  (  IN  At  IN  -)) 

No 

422, 

435, 

447, 

519 

424 

(S7  0  N7  (  LN  At  IN  LN)) 

No 

423, 

424, 

436. 

448, 

520 

426 

(S7  ()  N7  (LN  At)) 

No 

27  9. 

425, 

449, 

521 

426 

(S7  ()  N7  (LN  At  )) 

No 

425. 

426, 

450, 

522 

42/ 

(S7  0  N7  (LN  At  LN)) 

No 

426, 

427. 

451, 

523 

426 

(S7  0  N7  (IN  At  LN)) 

No 

425, 

428, 

452, 

524 

429 

(S7  0  N7  (LN  At  IN  )) 

No 

428, 

429, 

453. 

525 

430 

(S7  0  N7  UN  At  LN  IN)) 

No 

429,  430, 

454, 

526 

431 

(S7  ()  N7  (LN  LN  At)) 

No 

280, 

431 . 

455, 

52  7 

432 

(S7  ()  N7  (LN  LN  At  )) 

No 

431, 

432, 

456,  528 

433 

(S'7  ()  N7  (LN  IN  At  IN)) 

No 

432, 

433, 

457, 

529 

434 

(S'1  0  N7  UN  LN  At  LN)) 

No 

431, 

434, 

458, 

530 

43b 

(S7  ()  N7  (LN  LN  At  LN  )) 

No 

434, 

435, 

459. 

531 

436 

(S?  0  N7  (LN  LN  At  LN  LN)) 

No 

435, 

436, 

460, 

532 

43  7 

(S?  UN)  N7  (  At)) 

No 

281. 

413, 

43  7, 

449. 

533 

438 

(S7  UN)  N7  (  At  )) 

No 

414, 

437, 

438, 

450, 

534 

439 

(S7  (LN)  N7  (  At  LN)) 

No 

415, 

438, 

439, 

451, 

535 

440 

(S'7  (IN)  N?  (  At  IN)) 

No 

416, 

437, 

440. 

452. 

536 

441 

(S?  (LN)  N?  (  At  IN  )) 

No 

417, 

440, 

441, 

453, 

53  7 

442 

(S7  UN)  N?  (  At  IN  IN)) 

No 

418. 

441, 

442. 

454, 

538 

443 

(S?  (IN)  N?  (  IN  At)) 

No 

282, 

419, 

443, 

455, 

539 

4  14 

(S'7  (LN)  N7  (  LN  At  )) 

No 

420, 

C43, 

444, 

456, 

540 

44  b 

(S?  (LN)  N?  (  LN  A*  IN)) 

No 

421, 

444, 

445f 

457. 

541 

4  16 

( S 7  (LN)  N?  (  LN  At  INV 

No 

422, 

443, 

446, 

4  58. 

542 

44  7 

(S7  (IN)  N7  <  IN  At  IN  )) 

No 

423, 

446, 

447. 

456, 

543 

448 

(S?  (LN)  N?  (  LN  At  LN  LN)) 

No 

424, 

447, 

448, 

460, 

544 

449 

<S7  (IN)  N7  (IN  -  At)) 

No 

283, 

425. 

449, 

545 

460 

(S?  (LN)  N?  UN  -  At  )) 

No 

426, 

449, 

450, 

546 
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Is  Link 

Numtnrr  l  ink  Condition 

In  Tree?  Successors 

46 1 

(S?  (IN)  N?  (IN  A*  IN)) 

No 

42/, 

460, 

451, 

64  7 

462. 

(S?  (IN)  N?  (LN  A*  IN)) 

No 

428, 

449, 

452, 

548 

463. 

(S?  (IN)  N?  (IN  A*  IN  )) 

No 

429, 

462. 

453, 

549 

464 

(S?  (IN)  N?  UN  A*  IN  IN)) 

No 

430, 

453, 

464, 

550 

466 

(S?  (IN)  (IN  IN  A*)) 

No 

284, 

431, 

456, 

551 

460 

(S'7  (IN)  N-7  (IN  IN  A ♦  )) 

No 

432, 

465. 

456, 

552 

467 

(S?  UN)  UN  IN  A*  IN)) 

No 

433, 

456, 

457, 

653 

458 

( S' 7  (IN)  N?  UN  IN  A*  IN)) 

No 

434, 

456, 

458. 

554 

468 

(S’*  (LN)  H?  UN  LN  A*  LN  )) 

No 

435, 

468, 

459, 

655 

400 

( S' *  (IN)  N?  (IN  LN  A*  IN  IN)) 

No 

436, 

469, 

460, 

556 

401 

(S’*  ()  S  (♦  A*)) 

Yes 

286, 

473, 

485 

462 

(S?  ()  S  (♦  A+  )) 

Yes 

401, 

4/4, 

486 

403. 

(S’*  ()  S  (♦  A*  IN)) 

Yes 

402, 

40  3, 

4/5, 

487 

404 

( S ?  ()  S  (♦  A*  LN)) 

Yes 

461, 

464, 

4/6, 

488 

406 

( S' 7  ()  S  (♦  A*  IN  -)) 

Yes 

404, 

4  7  7. 

489 

400 

(S'7  ()  S  (♦  A*  IN  IN)) 

ves 

465, 

466. 

4/8, 

490 

40/ 

(S?  ()  S  (♦  LN  A*)) 

Yes 

286, 

4/9, 

491 

468 

( S' 7  ()  S  (♦  IN  A>  -)) 

Yos 

46/, 

480, 

492 

468 

(S?  ()  S  (♦  LN  A*  IN)) 

Yes 

468, 

409, 

481 . 

493 

4/0 

( S' *  ()  S  (♦  IN  A*  IN)) 

Yes 

46/, 

4/0, 

482, 

494 

4/1 

(S'*  0  S  (♦  IN  A*  IN  )) 

Yes 

4/0, 

483, 

495 

4/2 

(S?  ()  S  (♦  IN  A ♦  IN  IN)) 

Yes 

4/1. 

4/2, 

484, 

496 

4/3 

(S'7  ()  S  (IN  *  A*)) 

Yes 

28/. 

4/3, 

497 

4/4 

(S?  ()  S  (IN  *  A*  )) 

Yes 

4/3, 

4/4, 

498 

4/6 

(S’*  ()  S  (IN  *  A*  IN)) 

Yes 

4/4, 

4/6, 

499 

4/0 

(S'7  ()  S  (LN  *  A*  IN)) 

Yes 

4/3, 

4/6, 

500 

4// 

(S?  ()  S  (LN  ♦  A*  IN  -)) 

Yes 

4/6, 

4//, 

501 

4/8 

(S?  ()  S  (LN  *  A*  IN  -  IN)) 

Yes 

4  77, 

4/8, 

502 

478 

(S’*  <)  S  (LN  ♦  LN  A*)) 

Yes 

288. 

4/9, 

503 

480 

(S?  ()  S  (IN  ♦  IN  A*  )) 

Yes 

4/9, 

480, 

504 

48 1 

(S’*  ()  S  (LN  ♦  LN  A*  LN)) 

Yes 

480, 

481  , 

505 

482 

(S?  ()  S  (LN  ♦  LN  A ♦  LN)) 

Yes 

4/9, 

482, 

506 

483 

(S’*  ()  S  (IN  ♦  IN  A.  IN  )) 

Yes 

482, 

483, 

507 

484 

(S’*  ()  S  (LN  *  IN  A*  LN  IN)) 

Yes 

483, 

484, 

608 

485 

(S’*  (LN)  S  (♦  A*)) 

Ves 

289, 

461, 

485, 

49  7 

486 

(S’*  (IN)  S  (*  A*  )) 

Yes 

462, 

485, 

480, 

498 

48/ 

(S?  (LN)  S  (*  A*  IN)) 

Yes 

463, 

486, 

48/. 

499 

488 

(S’*  (IN)  S  (♦  A*  tN)) 

Yos 

464, 

486, 

488, 

600 

489 

(S?  (LN)  S  (♦  A*  LN  )) 

Yes 

465 

488, 

489, 

601 

490 

(S’*  (IN)  S  (♦  A*  IN  LN)) 

Yes 

466, 

489, 

490, 

602 

491 

( S’ *  (IN)  S  (♦  LN  A*)) 

Yes 

290, 

467, 

491 . 

603 

492 

(S'7  (IN)  S  (*  IN  A*  )) 

Yes 

408, 

491, 

492, 

604 

493 

(S?  (IN)  S  (♦  IN  A*  IN)) 

Yes 

409. 

492. 

493, 

605 

494 

(S’*  (IN)  S  (♦  LN  A*  LN)) 

Yes 

4/0, 

491, 

494, 

506 

495 

(S?  (LN)  S  (♦  LN  A*  IN  )) 

Ves 

4/1, 

494, 

495, 

50/ 

496 

(S’*  (IN)  S  (*  LN  A*  IN  LN)) 

Ves 

4/2. 

496, 

496. 

508 

49/. 

(S'*  (IN)  S  (IN  ♦  A*)) 

Yes 

291, 

4/3. 

49/ 

498 

(S?  (LN)  S  (IN  *  A*  )) 

Yes 

4/4, 

497, 

49b 

499 

(S?  (LN)  S  (  N  ♦  A*  -  LN)) 

Yes 

4/5, 

498. 

499 

600 

(S?  (IN)  S  (IN  4  A ♦  IN)) 

Yes 

4/6, 

497, 

500 
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Table  B.l.  Membership  Protocol  Link,  Conditions 


Zs  Z/nA 

Number  l  ink  Condition 

Zn  Tree?  Successor s 

601 

(S?  (LN)  S  UN  ♦  A*  LN  -)) 

Yes 

477. 

500, 

601 

b  02. 

(S?  UN)  S  UN  ♦  A*  LN  LN)) 

Yes 

4/8, 

501.  502 

603 

(S?  {LN)  S  UN  ♦  IN  A»)) 

Yes 

292, 

4/9, 

603 

604 

(S3  {LN)  S  (LN  ♦  LN  A*  >) 

Yes 

480, 

603, 

604 

606 

(S'*  (Z N)  S  {IN  *  IN  k*  LN)) 

Yes 

481 , 

604, 

605 

606 

(S'*  UN)  S  UN  *  LN  A*  LN)) 

Yes 

482, 

503, 

506 

60/ 

(S’  {LN)  S  {LN  *  LN  A*  IN  )) 

Yes 

483, 

606, 

50  7 

603 

(S?  UN)  S  {IN  *  IN  A*  LN  -  LN)) 

Yes 

484, 

507, 

608 

609 

(S’  ()  X’  (  A*)) 

No 

293, 

413, 

533 

610 

(S?  ()  X?  (  A*  -)) 

No 

4 1  4^ 

509, 

634 

61  1 

(S’  ()  X’  (-  A*  IN)) 

No 

415, 

510, 

51  1, 

535 

6 1  2 

(S’  ()  X’  (  At  LN)) 

No 

416, 

509, 

612, 

536 

613 

(S’  ()  X?  (  k*  LN  -)) 

No 

41  /. 

612, 

63/ 

61  4 

(S?  ()  X’  (  A*  LN  LN)) 

No 

418, 

613, 

514, 

638 

616 

(S?  <)  X?  (-  LN  A*)) 

No 

294, 

419, 

639 

616 

(S’  ()  X’  (-  IN  A*  -)) 

No 

420, 

616, 

640 

61  7. 

(S’  ()  X?  (  LN  k*  LN)) 

No 

421, 

516, 

61  7, 

541 

618. 

(S’  ()  X?  (-  IN  k*  IN)) 

No 

422, 

515, 

518, 

642 

619 

(S’  ()  X?  (  LN  k*  LN  )) 

No 

423, 

518, 

543 

620 

(S’  ()  X’  (  LN  k*  IN  LN)) 

No 

424, 

519, 

520, 

544 

63  1 

(S’  0  X?  {LN  A*)) 

No 

295, 

425, 

545 

622. 

(S’  ()  X?  UN  k*  )) 

No 

426, 

621. 

546 

533 

(S?  ()  X?  UN  -  k*  IN)) 

No 

42/. 

522, 

523, 

54/ 

634 

(S’  ()  X?  {LN  -  k*  LN)) 

No 

428, 

521, 

524. 

548 

526 

(S’  ()  X’  {IN  A ♦  LN  )) 

No 

429, 

524, 

549 

536 

(S’  ()  X?  {LN  -  A*  LN  LN)) 

No 

430, 

525, 

526, 

550 

62/ 

(S?  ()  X?  UN  ZN  A*)) 

No 

296, 

431. 

561 

628 

(S’  ()  X?  {IN  LN  A ♦  -)) 

No 

432, 

627, 

552 

620 

(S’  ()  X?  UN  -  LN  k*  -  LN)) 

No 

433, 

528, 

529, 

553 

63  C. 

(S?  ()  X?  {LN  LN  k*  LN') 

No 

434, 

62  7, 

530, 

554 

63  1 

(S’  ()  X’  UN  ZN  A*  LN  )) 

No 

436, 

530, 

555 

532 

(S’  ()  X?  {LN  LN  k*  LN  LN)) 

No 

436, 

531, 

532, 

556 

533 

(S’  {IN)  X?  (  A*)) 

No 

29/, 

366, 

389, 

43  7, 

533 

534 

(S’  {LN)  X’  (-  A*  )) 

No 

366, 

390, 

438, 

533, 

634 

635 

(S’  {LN)  X?  (  k*  LN)) 

No 

36/, 

391, 

439, 

534, 

535 

536 

(S’  UN)  X?  (  A+  IN)) 

No 

368, 

392, 

440. 

533,  536 

63/ 

(S’  UN)  X?  (  A*  LN  -)) 

No 

369. 

393, 

441, 

536, 

63/ 

638 

(S?  UN)  X’  (-  A*  IN  -  ZN)) 

No 

3/0, 

394, 

442, 

53  7, 

638 

539 

(S’  UN)  X?  (  ZN  A*)) 

No 

298, 

3/1, 

395, 

443, 

539 

640 

(S’  {IN)  X?  (  ZN  A*  -)) 

No 

3/2, 

396, 

444, 

539, 

640 

541 

(S?  {LN)  X?  (  IN  A*  -  Z  N)) 

No 

3/3, 

397, 

445, 

540, 

541 

642 

(S’  {IN)  X’  (  ZN  A*  ZN)) 

No 

3/4, 

398. 

446, 

639, 

642 

643 

(S’  (ZN)  X?  (-  ZN  A»  ZN  )) 

No 

3/5, 

399, 

44,  , 

542, 

643 

544 

(S’  (ZN)  X?  (  ZN  A*  ZN  iN)) 

No 

3/6, 

400, 

448, 

643, 

544 

64  5 

(S’  (ZN)  X’  (ZN  A*;) 

No 

299, 

3  7  7, 

401, 

449. 

545 

646 

(S’  (ZN)  X’  (ZN  A*  )) 

No 

3/8, 

402. 

450, 

545, 

546 

64/ 

(S’  (Z.V)  X’  (ZN  -  A*  -  ZN)) 

No 

3/9, 

403, 

461, 

646,  64/ 

648 

(S?  (ZN)  X?  (ZN  -  A^  ZN)) 

No 

380, 

404, 

452, 

545, 

648 

649 

(S’  (ZN)  X?  (ZN  -  A*  ZN  -)) 

No 

381, 

405, 

453, 

548, 

549 

560 

(S’  (ZN)  X?  (ZN  -  A»  ZN  ZN)) 

No 

382, 

406, 

454, 

649.  660 

2 44  Appendix  B:  Correctness  of  the  Membership  Protocol 


Table  ti.l:  Membership  Protocol  link  Conditions 


Number  l  ink  Condition 


651  (S'*  UN)  X?  UN  IN  A+)) 

552.  (S7  UN)  X?  UN  LN  A*  )) 
653  (S7  UN)  X?  UN  IN  k*  IN)) 

554.  (S?  UN)  X"»  (//V  FN  A^  /A/)) 
666.  (S?  UN)  W  UN  IN  k*  IN  -)) 
556.  (S?  (/AT)  X?  UN  -  IN  k+  IN  I 
65/.  (X  ()  M?  (♦)) 

558  (X  ()  M?  (/A/  ♦)) 

559  (X  0  N  ()) 

560  (X  ()  X  ()) 


Is  Link 

In  Tree?  Successors 

No 

300, 

383, 

407. 

455, 

551 

No 

384, 

408, 

456, 

651, 

552 

No 

386, 

409, 

46/, 

552, 

663 

No 

386, 

410, 

458, 

551, 

664 

No 

38/, 

411, 

460, 

664, 

666 

No 

388, 

412. 

460, 

655, 

556 

No 

222, 

265, 

558 

No 

223, 

266, 

558 

No 

224, 

56/, 

560 

No 

569 

We  can  prove  directly  from  Table  B  1  that  our  protocol  has  the  desired  closure 


and  consistency  properties  alluded  to  at  the  beginning  ot  this  section  for  closure, 
we  note  that  there  is  no  configuration  in  the  table  In  which  a  processor  is 
presented  with  a  message  it  cannot  handle  in  its  current  state  For  consistency, 
we  note  that  the  only  quiescent  link  conditions  are  our  original  five 


(X  ()  X  ())  (X  ()  N  ())  (MONO)  (L  ()  L  0)  (S  ()  M  ()) 


and  that  each  of  these  has  the  desired  status  of  the  link  as  being  in  or  out  of  the 


reference  tree 


B.2:  Global  Properties  of  the  Membership  Protocol 

Determining  additional  properties  of  the  membership  protocol  requires  combining 
the  data  presented  in  this  section  with  a  little  graph  theory.  For  our  purposes,  we 
make  the  following  definitions 
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Definition  B.2: 


An  undirected  graph  is  an  ordered  pair  ( N,A ),  where  N  is  a  subset  ol  the  set  N 
of  nodes,  and  A  is  a  set  of  arcs  { m.n }  such  that  m.n  £  N. 

Intuitively,  for  every  arc  drawn  between  nodes  m  and  n  in  an  undirected  graph 
(N.A).  A  contains  a  doubleton  set  (m.r?) 

Definition  B.3: 

A  node  n  is  a  leal  node  of  the  undirected  graph  G  =  (A/. 4)  iff  n  t  N  and  there  is 
exactly  one  node  n‘  such  that  { n.n '}  t  A  A  node  n  t  N  is  an  isolated  node  of  G  iff 

n  4  a  for  any  a  £  A 

The  concept  of  a  leaf  node  is  the  familiar  one  of  a  node  with  exactly  one  arc 
attached,  an  isolated  node  is  a  node  with  no  arcs  attached  to  it 

Definition  B.4: 

A  reference  tree  transformation  of  an  undirected  graph  G  =  {H.A)  is  one  of 

1.  the  original  graph  G,  or 

2.  a  new  graph  G'  =  (Vv{n}./1  {{m.n}})  whore  m  and  n  are  nodes  such  that 
m  t  N  and  n  4  N ,  or 

3  a  new  graph  G'  =  {M  {n}  .A- { {m.n} })  where  m.n  f  A/,  n  is  a  leal  node  of 
G,  and  {m,n}  £  A. 

Intuitively,  a  reference  tree  transformation  of  a  graph  G  is,  if  G  is  expanded,  a 
graph  with  a  new  node  such  that  the  new  node  is  a  leaf  node  and  is  connected  to 
some  other  node  already  in  G,  and  if  G  is  contracted,  c  ne>v  graph  minus  one  of  the 
leaf  nodes  in  G  (and  also  minus  the  associated  arc)  The  purpose  of  defining  refer¬ 
ence  tree  transformations  is  to  introduce 
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Lemma  B.5: 


If  the  set  of  processors  and  links  that  are  members  of  a  reference  tree  is 
viewed  as  an  undirected  graph,  the  membership  protocol  performs  only  reference 
tree  transformations  on  reference  trees 


proof  follows  from  the  Information  presented  in  Table  B  1,  with  some  elaborations. 
Our  strategy  is  to  examine  the  transformation  effected  by  each  kind  of  transition 
from  onu  link  condition  to  another 

case  1  No  change  occurs  in  the  membership  status  of  processors  or  links  This 
is  a  valid  reference  tree  transformation,  by  clause  1  of  Definition  B.4. 

case  ?  One  of  the  processors  undergoes  the  transition  X:R  +  /L  +  :S  or  X?:/W/:N! 
In  this  case  both  the  link  and  the  processor  undergoing  the  transition  join 
the  reference  tree  In  all  cases  where  this  happens,  the  processor  at  the 

other  end  of  the  link  is  already  in  the  reference  tree,  making  this  an 

instance  of  clause*  ?  of  Definition  B  4,  assuming  that  no  other  links  join  the 
reference  tree  at  the  same  time  This  assumption  is  valid  because  the 
other  link  states  of  the  processor  joining  the  reference  tree  will  simultane¬ 
ously  undergo  either  the  transition  X:/:N  or  the  transition  X?:/:N?,  neither  of 
which  causes  any  of  these  links  to  Join  the  reference  tree. 

case  3  One  of  the  processors  undergoes  the  transition  X:/:N  or  X?:/:N?.  In 
this  case  the  processor  joins  the  reference  tree  but  the  link  does  not.  If 
some  other  link  attached  to  the  processor  simultaneously  Joins  the  reference 
tiee,  as  in  case  P,  above,  then  the  resulting  transformation  is  a  valid  refer¬ 
ence  tree  transformation  It  would  be  possible,  though,  to  imagine  a  situa- 

t'on  in  which  a  processor  spontaneously  decided  to  join  an  object's  refer¬ 
ence  tree,  say,  by  undergoing  the  transition  X:/:N  tor  all  attached  links 
This  would  not  be  a  reference  tree  transformation  because  a  processor 
would  join  the  "tree"  without  any  corresponding  link  also  Joining.  This 
scenario  is  ruled  out,  however,  by  the  restrictions  on  the  applicability  of 
these  transitions  discussed  at  the  head  of  this  appendix. 

case  4  One  of  the  processors  undergoes  the  transition  M:/-:X?.  In  this  case 
both  the  link  and  the  processor  undergong  the  transition  leave  the  refer¬ 
ence  tree  This  transition  is  only  allowed  if  the  processor's  state  for  every 
other  link  is  either  N  or  N?,  which  (as  can  be  determined  from  examination 
of  Table  B  1)  ensures  that  no  other  link  attached  to  that  processor  is  in  the 
reference  tree  Hence  the  processor  undergoing  the  transition  is  a  leaf 
node  of  the  reference  tree,  making  this  a  reference  tree  transformation  of 
the  kind  described  in  clause  3  of  Definition  B.4. 

case  5:  One  of  the  processors  undergoes  the  transition  N:/:X  or  N?:/:X?.  This 
can  only  happen  If  on  some  other  link  the  processor  is  undergoing  a  transi¬ 
tion  of  *he  type  discussed  in  case  4.  Therefore  this  case,  too,  satisfies 
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clause  3  of  Definition  B  4. 


These  five  cases  exhaust  the  k'nds  of  changes  that  occur  between  link  conditions 
that  appear  as  successors  of  each  other  in  Table  B  1 

□ 


We  now  proceed  to  develop  some  more  terminology 

Definition  B.6: 

A  path  from  node  n  to  node  n'  in  an  undirected  graph  G  -  (A/. 4)  is  a  set 
P  -  {aQ.a^,  ,a^)  t  4  such  that 

1.  there  exists  a  sequence  ^  such  that  for  0  £  /  S  A, 

ai  *  imrml+0' 

2.  if  /  *  J.  then  a{  *  a^;  and 

3.  rr>Q  =  n  and  1  =  ri 

A  path  between  two  nodes  is  thus  a  set  of  arcs,  with  no  arc  used  more  than  once, 
which  connects  the  two  nodes 

Definition  B.7: 

An  undirected  graph  G  =  (A/,4)  is  connected  iff  for  any  m,n  *  N.  whore  m  *  n, 
there  exists  a  path  from  m  to  n  in  G 

Definition  B.8: 

An  undirected  graph  G  =  (W.4)  is  acyclic  Iff  for  any  m,n  «  AT,  where  m  *  n,  there 
is  at  most  one  path  from  m  to  n  in  G 
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Some  subsidiary  lemmas  help  support  the  main  result  of  this  section 


Lemma  B.9: 

If  an  undirected  graph  G  -  (A/, A)  has  the  property  that  | A/ 1  =  tfien  either 

G  has  a  leaf  node  or  G  has  an  isolated  node. 


proof  by  contradiction  Assume  G  has  no  leaf  node  or  isolated  node.  Then  tor 
every  node  n  t  N,  there  are  at  least  two  arcs  a  f  fl  such  that  n  t-  a  Since  all  arcs 
are  doubleton  sets,  the  total  number  of  arcs  must  be  at  least  as  great  as  the  total 
number  of  nodes  Thus  |A/|  s  |d|,  contradicting  the  assumption  that  [ A/ 1  =  1+|rT[. 


Lemma  B.  1 0: 

If  an  undirected  graph  G  -  (A/, A)  is  connected  and  |A/|  =  1+|l3|,  then  the  graph  G 
is  acyclic 


proof  by  induction  on  |/!| 

basis  |/1|  =  0  Then  there  is  only  one  node  in  G  and  Definition  B.8  is  satisfied 
vacuously 

irxluction  Assume  the  lei.ima  for  |4|  =  A  If  j>T|  =  A*  1  then  G  has  at  least  two 
nodes  Since  G  is  connected,  there  must  be  a  path  between  any  node  in  G 
and  any  other  No  none  In  G  can  be  an  isolated  node,  for  there  ran  be  no 
path  from  an  isolated  node  to  any  other  node  Therefore,  by  lemma  B  9,  G 
must  have  at  least  one  leaf  node  n  Let  {m.n)  e  A  Lie  the  one  arc  connect¬ 
ing  to  node  n.  Then  the  graph  G'  =  (N  {n).A  {{m.n)))  is  the  graph  G  minus 
the  leaf  node  n  and  its  connecting  arc  By  the  induction  hypothesis,  G'  is 
acyclic.  Since  n  is  a  leaf  node  of  G,  the  only  paths  in  G  that  include  the  arc 
{ m.n }  are  those  that  start  or  end  on  n.  In  order  for  G  to  be  acyclic,  there 
must  be  at  most  one  path  between  any  pair  /.A.  of  distinct  nodes  in  G.  Two 
cases  can  be  distinguished. 

case  1  /,A  f  G'  Then  ;  *  n  and  A  *  n.  hence  no  path  from  y  to  A  in  G  could 
include  the  arc  {m.n')  Thus  any  path  between  y  and  A  in  G  is  also  a  path 
between  j  and  A  in  G'  By  the  induction  hypothesis,  there  can  he  at  most 
one  of  these 

case  ?  j  -  n  or  A  -  r.  Without  loss  of  generality,  we  can  assume  that  /  =  n 
The. i  any  path  from  /  to  A  in  G  must  beg'n  with  the  arc  {m.n)  If  A  =  m,  this 
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arc  is  the  unique  path  from  /  to  A  It  A  t  m,  a  path  from  /  to  A  must  be  the 
union  of  { m.n }  and  a  path  from  m  to  A  in  G  Since  m.A  t  G',  any  path  from  m 
to  A  in  G  Is  also  In  G'  By  the  induction  hypothesis,  there  is  at  most  ono 
path  from  m  to  A  in  G'  Hence  there  is  at  most  one  path  from  ;  to  A  in  G 


□ 


Lemma  B.  1  1 : 

If  a  graph  G  =  (AM)  has  the  property  that  | A/|  =  1  then  any  reference  tree 
transformation  G'  =  (A/-./)')  of  G  has  the  property  that  |A/'|  =  1  |/1"| 

proof  Fach  of  the  throe  casus  of  Definition  B  1  adds  or  subtracts  the  same 
number  of  elements  to  or  fiom  N  and  A,  tnus  Keeping  constant  the  difference 
betweon  their  cardinalities 


o 


Lemma  B.12: 

If  a  graph  G  =  (N.A)  is  connected,  then  any  reference  tree  transformation 
G'  ~  (AT.4')  of  G  Is  also  connected. 

proof  by  cases 

case  1  G'  =  G  Then  G’  is  obviously  connected  If  G  is 

case  2  G'  =  (AA'{n},A  {{m.n)})  where  m  and  n  are  nodes  such  that  m  t  N  and 
n  4  A/  Then  G’  is  connected  if  there  exists  a  path  in  G'  between  every  two 
distinct  nodes  /.A  in  G'  If  /.A  *  N  then  there  exists  a  path  from  /  to  A  using 
only  arcs  in  G,  all  of  which  are  also  present  in  G'  Otherwise  either  j  =  n  or 
A  -  n  Without  loss  of  generality  we  may  assume  /  <-  n  Then,  if  A  -  m, 
{{m.n})  Is  a  path  from  /  to  A  in  G'  Otherwise,  If  A*  Is  a  path  from  m  to  A  in 
G,  then  /Ai{{m,n}}  is  a  path  from  /  to  A  in  G  Hence  G'  is  connected  if  G  is 

case  3-  G'  =  (A/  {n},4  {{m.n)})  where  m.n  »  N,  n  is  a  leaf  node  of  G,  and 
{m.n)  t  A  Since  n  is  a  leaf  node  of  G.  no  path  between  nodes  /  and  A  in  G. 
where  j  *  n  and  A  t  n,  could  Include  the  ari  {m  n)  Thus  any  such  path 
between  nodes  j  and  A  in  G  is  also  j  path  between  j  and  A  in  G'  Since  any 
pair  of  nodes  in  G'  falls  into  this  category,  and  since  G  is  connected,  G'  must 
be  connected 


□ 
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Lemma  B.13. 


If  a  graph  G  -  (N.A)  is  connected  and  has  the  property  that  |N|  =  1*1/11,  the 
result  W  *  (N'.A')  of  performing  any  number  of  successive  reference  tree  transfor¬ 
mations  on  G  is  also  connected  and  has  the  property  that  |W|  =  1*|A'| 


proof  by  induction  on  the  number  A  of  reference  tree  transformations  applied 
bas/s  A  =  0  Then  G  -  G'  and  the  lemma  is  trivially  true 

induction  Assume  the  lemma  for  A  =  n  successive  reference  tree  transforma¬ 
tions  To  show  the  result  for  A*1  transformations,  consider  the  graph 
G "  =  (A/", A")  resulting  at  the  end  of  the  first  A  transformations  in  the 
sequence  By  the  hypothesis,  G"  is  connected  and  has  the  property  that 
|A/“|  =  1*|A"|  Then  one  more  transformation  on  G“  will  produce  G’  By  lem¬ 
mas  B  1  1  and  B  1?.  G'  is  connected  and  has  the  property  that  |A/'|  s  1»|4'| 


□ 


Theorem  B.14: 

Any  number  of  successive  reference  tree  transformations  on  the  Initial  graph 
G  *  ({n).{})  will  always  yield  a  connected,  acyclic  graph 


proof  G  *  (A /.A)  Is  connected  and  has  the  property  that  |A/|  =  1  *|4|  Thus  by 
lemma  B  13  any  graph  G'  *  (N'.A)  produced  from  G  by  any  number  of  successive 
reference  tree  transformations  is  connected  and  has  the  property  that  |Af|  *  1*|A'| 
By  Lemma  B  10,  this  means  that  G'  is  acyclic  as  well  as  being  connected 


□ 

Since  every  object's  reference  tree  starts  out  including  just  the  single  proces¬ 
sor  where  the  object  was  created,  Theorem  B  14  assures  us  that  no  reference 
tree  wl'l  ever  become  disconnected  or  come  to  include  cycles 
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Section  B .2 


261. 


B.3:  Summary 

This  appendix  first  recapitulated  the  workings  ot  the  reference  tree  member 
ship  protocol,  including  the  various  restrictions  on  the  applicability  of  some  of  the 
state  transitions  This  protocol  was  then  analyzed  tor  the  local  properties  ot  clo¬ 
sure  and  consistency,  and  the  global  properties  of  resistance  to  disconnection  and 
resistance  to  forming  cycles  Conclusions  of  the  analysis  were  as  follows 

•  closure  every  message  that  can  arrive  at  a  processor  In  a  particular  state 
can  be  handled  by  the  processor  In  that  state. 

•  consistency —  our  assignment  of  link  status  (as  being  in  or  out  of  the  refer¬ 
ence  troe)  to  link  conditions  is  consistent  both  with  our  assumptions  about 
when  processors  entor  and  leave  the  tree  and  with  the  link  status  desired 
to  accompany  each  quiescent  link  condition  (condition  In  which  no  messages 
are  In  transit) 

a  resistance  to  disconnection  —  reference  trees  do  not  become  disconnected. 

•  resistance  to  forming  cycles  —  cycles  do  not  form  In  reference  trees 
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